Oct 31 18:20:05.746445 ip-10-0-140-47 systemd[1]: Starting Kubernetes Kubelet... Oct 31 18:20:06.118432 ip-10-0-140-47 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 18:20:06.118432 ip-10-0-140-47 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 18:20:06.118432 ip-10-0-140-47 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 18:20:06.118432 ip-10-0-140-47 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 18:20:06.125075 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.124882 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.125075 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125064 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.125075 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125071 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.125075 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125075 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.125075 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125080 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.125075 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125084 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125088 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125092 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125096 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125099 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125103 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125108 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125112 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125115 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125118 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125122 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125125 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125129 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125133 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125137 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125141 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125144 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125148 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125152 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.125456 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125155 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125159 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125163 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125168 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125175 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125198 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125203 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125207 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125218 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125222 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125226 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125230 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125234 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125238 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125242 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125247 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125251 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125256 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125260 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.126161 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125264 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125269 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125273 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125277 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125281 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125286 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125290 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125294 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125298 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125303 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125307 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125311 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125318 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125322 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125326 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125330 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125334 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125338 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125342 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125346 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.126616 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125350 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125354 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125358 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125363 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125367 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125371 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125374 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125378 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125383 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125388 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125395 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125402 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125407 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125412 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125416 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125420 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125425 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125429 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125433 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.127145 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125437 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125450 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125455 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125459 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125463 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125466 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125471 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125475 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125479 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125483 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125487 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125491 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125495 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125499 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125503 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125507 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125513 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125517 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125521 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125525 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.127605 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125530 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.128327 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125535 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.128327 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125539 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.128327 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.125550 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.128629 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128612 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.128629 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128629 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128634 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128639 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128643 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128647 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128651 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128655 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128659 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128663 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128675 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128679 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128683 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128687 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128691 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128695 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128699 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128703 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128707 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128712 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128716 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.128740 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128720 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128724 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128728 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128732 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128736 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128740 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128744 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128748 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128753 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128756 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128760 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128764 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128768 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128774 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128781 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128785 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128789 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128793 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128797 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.129587 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128801 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128805 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128809 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128821 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128825 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128829 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128833 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128838 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128842 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128846 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128850 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128854 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128858 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128862 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128866 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128871 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128876 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128880 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128884 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.130381 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128888 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128892 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128897 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128901 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128905 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128909 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128913 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128917 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128922 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128944 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128950 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128955 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128959 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128963 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128967 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128972 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128976 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128987 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128992 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.128996 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.130980 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129000 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129005 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129010 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129014 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129021 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129027 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129032 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129037 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129041 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129045 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129049 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129053 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129057 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129060 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129064 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129069 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129073 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129077 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129081 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.131501 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129086 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129090 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129094 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129097 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129101 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129105 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129109 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.129114 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130092 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130105 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130110 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130115 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130119 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130124 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130130 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130134 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130139 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130143 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130147 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130151 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.132169 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130156 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130160 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130164 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130168 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130172 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130176 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130198 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130203 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130207 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130210 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130215 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130219 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130223 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130227 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130231 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130235 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130239 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130243 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130247 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.133011 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130251 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130255 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130259 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130264 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130269 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130273 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130277 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130282 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130287 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130291 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130295 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130299 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130303 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130308 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130316 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130320 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130324 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130328 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130332 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130336 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.133827 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130340 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130344 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130348 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130352 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130356 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130361 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130365 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130369 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130373 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130376 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130380 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130384 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130388 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130391 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130395 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130399 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130403 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130407 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130412 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130417 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.134348 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130421 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130425 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130431 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130436 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130442 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130447 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130452 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130457 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130462 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130466 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130470 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130474 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130482 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130489 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130494 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130499 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130503 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130508 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.134966 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130514 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130519 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130524 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130529 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130534 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130538 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130543 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130549 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130553 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130557 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130561 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130565 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130570 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130574 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130579 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130583 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.130588 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131648 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131665 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131675 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131682 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 18:20:06.135531 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131689 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131694 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131701 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131708 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131713 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131717 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131723 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131728 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131733 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131737 2572 flags.go:64] FLAG: --cgroup-root="" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131742 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131746 2572 flags.go:64] FLAG: --client-ca-file="" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131751 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131755 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131763 2572 flags.go:64] FLAG: --cluster-domain="" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131767 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131772 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131777 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131782 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131789 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131794 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131798 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131803 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131808 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 31 18:20:06.136013 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131815 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131819 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131824 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131829 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131835 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131840 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131845 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131849 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131854 2572 flags.go:64] FLAG: --enable-server="true" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131858 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131865 2572 flags.go:64] FLAG: --event-burst="100" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131870 2572 flags.go:64] FLAG: --event-qps="50" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131874 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131879 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131883 2572 flags.go:64] FLAG: --eviction-hard="" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131890 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131895 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131899 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131904 2572 flags.go:64] FLAG: --eviction-soft="" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131909 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131914 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131918 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131923 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131927 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131931 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 18:20:06.136738 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131936 2572 flags.go:64] FLAG: --feature-gates="" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131942 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131947 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131952 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131957 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131961 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131966 2572 flags.go:64] FLAG: --help="false" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131974 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131981 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131986 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131990 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.131996 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132001 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132006 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132010 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132014 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132024 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132029 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132035 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132039 2572 flags.go:64] FLAG: --kube-reserved="" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132044 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132048 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132053 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 18:20:06.137470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132057 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132063 2572 flags.go:64] FLAG: --lock-file="" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132068 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132072 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132077 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132086 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132090 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132094 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132099 2572 flags.go:64] FLAG: --logging-format="text" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132104 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132110 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132114 2572 flags.go:64] FLAG: --manifest-url="" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132118 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132125 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132130 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132135 2572 flags.go:64] FLAG: --max-pods="110" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132144 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132149 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132161 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132166 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132170 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132174 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132195 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132208 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132212 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 18:20:06.138022 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132217 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132221 2572 flags.go:64] FLAG: --pod-cidr="" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132226 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132231 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132235 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132240 2572 flags.go:64] FLAG: --port="10250" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132244 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132249 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-076b9146017615332" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132254 2572 flags.go:64] FLAG: --qos-reserved="" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132258 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132263 2572 flags.go:64] FLAG: --register-node="true" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132268 2572 flags.go:64] FLAG: --register-with-taints="" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132274 2572 flags.go:64] FLAG: --registry-burst="10" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132279 2572 flags.go:64] FLAG: --registry-qps="5" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132283 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132288 2572 flags.go:64] FLAG: --reserved-memory="" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132293 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132298 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132303 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132308 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132312 2572 flags.go:64] FLAG: --runonce="false" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132317 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132322 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132330 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132334 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132338 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 18:20:06.138602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132345 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132350 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132355 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132360 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132364 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132369 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132373 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132378 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132382 2572 flags.go:64] FLAG: --system-cgroups="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132387 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132395 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132400 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132404 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132411 2572 flags.go:64] FLAG: --tls-min-version="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132416 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132420 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132425 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132429 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132434 2572 flags.go:64] FLAG: --v="2" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132441 2572 flags.go:64] FLAG: --version="false" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132447 2572 flags.go:64] FLAG: --vmodule="" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132453 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.132458 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132681 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.139218 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132687 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132692 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132696 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132700 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132704 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132712 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132716 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132721 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132725 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132730 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132734 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132739 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132743 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132748 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132752 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132756 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132760 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132764 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132768 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.139769 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132772 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132776 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132780 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132785 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132789 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132793 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132797 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132801 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132805 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132809 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132813 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132817 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132822 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132826 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132829 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132834 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132838 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132842 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132849 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132854 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.140214 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132858 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132862 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132866 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132871 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132875 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132879 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132883 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132887 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132891 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132895 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132900 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132904 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132908 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132912 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132916 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132920 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132924 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132928 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132932 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132936 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.140671 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132940 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132945 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132949 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132953 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132957 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132961 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132967 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132972 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132976 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132980 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132987 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132991 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132995 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.132999 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133003 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133007 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133012 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133016 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133020 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.141126 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133024 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133027 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133031 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133035 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133038 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133043 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133047 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133051 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133055 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133059 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133063 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133068 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133072 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133077 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133083 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133089 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133093 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133098 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133102 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.141691 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133106 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133110 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133114 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133118 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133125 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133129 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133133 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133137 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.133200 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133265 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133272 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133277 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133282 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133286 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133291 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133295 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.142120 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133299 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133303 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133307 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133311 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133315 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133319 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133323 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133327 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133331 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133335 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133339 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133343 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133348 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133352 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133356 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133360 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133364 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133368 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133372 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133376 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.142500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133382 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133389 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133393 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133397 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133402 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133406 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133410 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133414 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133418 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133422 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133426 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133430 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133434 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133438 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133442 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133446 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133450 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133454 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133458 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133462 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.142951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133466 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133470 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133474 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133479 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133483 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133487 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133491 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133495 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133499 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133503 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133508 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133514 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133521 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133528 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133532 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133536 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133540 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133550 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133555 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133559 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.143417 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133563 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133569 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133574 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133578 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133582 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133586 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133590 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133594 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133598 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133602 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133606 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133610 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133615 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133619 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133625 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133629 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133634 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133640 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133646 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.143874 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133650 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133654 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133658 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133663 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133666 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133672 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133678 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133683 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133687 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133691 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133695 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133699 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133703 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133707 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133711 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133716 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133720 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133724 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.144324 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133728 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.133734 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.143092 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.143110 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.143128 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.143132 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143210 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143216 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143220 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143223 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143226 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143229 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143232 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143235 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143237 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143240 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143242 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143245 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143248 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143250 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.144722 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143253 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143255 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143258 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143260 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143264 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143268 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143270 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143273 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143275 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143278 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143280 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143283 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143285 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143288 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143291 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143295 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143297 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143300 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143303 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.145168 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143305 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143308 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143310 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143313 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143315 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143318 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143321 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143323 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143326 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143328 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143330 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143333 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143335 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143338 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143341 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143343 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143346 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143348 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143351 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143353 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.145657 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143356 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143359 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143361 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143363 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143366 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143369 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143372 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143374 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143377 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143380 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143383 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143385 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143388 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143390 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143393 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143395 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143397 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143400 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143402 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143405 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.146097 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143407 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143410 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143412 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143414 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143417 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143419 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143422 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143426 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143431 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143434 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143437 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143440 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143443 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143445 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143447 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143450 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143453 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143456 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143458 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.146600 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143461 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143464 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143466 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143469 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143472 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143474 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143477 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143479 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143482 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143484 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143487 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143489 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143491 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143494 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.143521 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143683 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.147062 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143689 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143692 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143695 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143697 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143700 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143702 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143704 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143707 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143709 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143712 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143714 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143717 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143719 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143722 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143724 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143727 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143730 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143732 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143735 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.147445 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143737 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143740 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143742 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143745 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143747 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143749 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143752 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143754 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143757 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143759 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143761 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143764 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143766 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143768 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143771 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143773 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143776 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143778 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143781 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143783 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.147880 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143786 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143788 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143791 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143793 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143796 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143798 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143801 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143803 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143806 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143808 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143810 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143813 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143815 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143818 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143821 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143823 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143827 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143830 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143833 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.148362 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143835 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143838 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143840 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143843 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143845 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143847 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143850 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143852 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143855 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143857 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143860 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143862 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143865 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143867 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143870 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143872 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143875 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143878 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143880 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143883 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.148784 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143886 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143890 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143893 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143895 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143898 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143900 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143903 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143905 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143910 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143912 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143914 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143917 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143919 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143921 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143923 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143926 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143928 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143930 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143932 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143935 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.149318 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143937 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143940 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143942 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143944 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143947 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143949 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.143951 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.143973 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144002 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144006 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144008 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144011 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144013 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144015 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144019 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144021 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:20:06.149768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144024 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144026 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144029 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144031 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144033 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144036 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144038 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144041 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144043 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144045 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144048 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144050 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144052 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144055 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144057 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144060 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144064 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144067 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144070 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:20:06.150123 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144073 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144076 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144079 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144082 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144085 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144087 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144089 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144092 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144095 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144097 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144100 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144102 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144104 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144106 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144109 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144111 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144113 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144116 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144118 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144121 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:20:06.150569 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144123 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144125 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144129 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144132 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144135 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144137 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144139 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144142 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144144 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144148 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144150 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144153 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144155 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144157 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144159 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144162 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144164 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144167 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144169 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:20:06.151060 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144171 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144174 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144176 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144194 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144198 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144202 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144206 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144208 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144211 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144213 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144216 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144218 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144220 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144224 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144226 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144229 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144231 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144233 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144236 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:20:06.151500 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144238 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144240 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144243 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144249 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144252 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144254 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144258 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144260 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144262 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144265 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144270 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144272 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144275 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144277 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144280 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144282 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144285 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144287 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144290 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144292 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:20:06.152058 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:06.144294 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:20:06.152534 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.144349 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 18:20:06.152534 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.146648 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 18:20:06.152534 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.148152 2572 server.go:1014] "Starting client certificate rotation" Oct 31 18:20:06.152534 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.148248 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 18:20:06.152534 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.148275 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 18:20:06.152534 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.150836 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 18:20:06.156309 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.156289 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 18:20:06.169592 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.169565 2572 log.go:25] "Validated CRI v1 runtime API" Oct 31 18:20:06.173217 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.173202 2572 log.go:25] "Validated CRI v1 image API" Oct 31 18:20:06.174064 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.174037 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 18:20:06.179120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.179095 2572 fs.go:135] Filesystem UUIDs: map[4bc14f40-7879-4892-9345-58abe23559d1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 edc1c76c-d85d-4082-8bb1-61227a4498b0:/dev/nvme0n1p4] Oct 31 18:20:06.179212 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.179120 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 18:20:06.184483 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.184373 2572 manager.go:217] Machine: {Timestamp:2026-10-31 18:20:06.182964806 +0000 UTC m=+0.340765944 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3217281 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2a141c5a0550a5fee13bfa60519c SystemUUID:ec2d2a14-1c5a-0550-a5fe-e13bfa60519c BootID:5d45d1a3-b11e-4f82-9136-d3314b7fd47c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:ee:6e:d0:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:ee:6e:d0:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:64:48:02:ce:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 18:20:06.184483 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.184480 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 18:20:06.184612 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.184562 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 18:20:06.187220 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.187193 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 18:20:06.187350 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.187221 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-47.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 18:20:06.188067 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.188057 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 18:20:06.188095 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.188070 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 18:20:06.188120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.188101 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 18:20:06.188120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.188115 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 18:20:06.189395 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.189385 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 18:20:06.193712 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.193700 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 18:20:06.193834 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.193825 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 18:20:06.194685 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.194675 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 31 18:20:06.194714 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.194690 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 18:20:06.194714 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.194699 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 18:20:06.194714 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.194710 2572 kubelet.go:405] "Adding apiserver pod source" Oct 31 18:20:06.194828 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.194718 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 18:20:06.195106 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.195093 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 18:20:06.197523 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.197503 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 18:20:06.206150 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.206131 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 18:20:06.207046 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207033 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 18:20:06.207083 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207064 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 18:20:06.207281 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207266 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 18:20:06.207315 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207296 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 18:20:06.207315 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207310 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 18:20:06.207375 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207326 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 18:20:06.207375 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207335 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 18:20:06.207375 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207343 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 18:20:06.208114 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207923 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 18:20:06.208114 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207929 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 18:20:06.208114 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207937 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 18:20:06.208114 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207943 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 18:20:06.208114 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207951 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 18:20:06.208114 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.207960 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 18:20:06.208693 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.208682 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 18:20:06.208727 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.208695 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 18:20:06.212560 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.212547 2572 server.go:1263] "Started kubelet" Oct 31 18:20:06.212730 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.212622 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 18:20:06.212782 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.212672 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 18:20:06.212782 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.212764 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 18:20:06.213332 ip-10-0-140-47 systemd[1]: Started Kubernetes Kubelet. Oct 31 18:20:06.214290 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.214246 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 18:20:06.214901 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.214888 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 31 18:20:06.220548 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.220530 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 18:20:06.221250 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.221209 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 18:20:06.221797 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.221781 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 18:20:06.221909 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.221891 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:06.222558 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222543 2572 factory.go:55] Registering systemd factory Oct 31 18:20:06.222558 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222558 2572 factory.go:223] Registration of the systemd container factory successfully Oct 31 18:20:06.222710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222580 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 18:20:06.222710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222591 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 18:20:06.222710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222582 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 18:20:06.222710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222638 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 31 18:20:06.222710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222644 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 31 18:20:06.222866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222767 2572 factory.go:153] Registering CRI-O factory Oct 31 18:20:06.222866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222776 2572 factory.go:223] Registration of the crio container factory successfully Oct 31 18:20:06.222866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222817 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 18:20:06.222866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222835 2572 factory.go:103] Registering Raw factory Oct 31 18:20:06.222866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.222851 2572 manager.go:1199] Started watching for new ooms in manager Oct 31 18:20:06.223907 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.223892 2572 manager.go:319] Starting recovery of all containers Oct 31 18:20:06.233939 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.233919 2572 manager.go:324] Recovery completed Oct 31 18:20:06.236074 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.236044 2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 18:20:06.238870 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.238858 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.240347 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240330 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.240407 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240362 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.240407 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240372 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.240828 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240815 2572 cpu_manager.go:226] "Starting" policy="none" Oct 31 18:20:06.240869 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240827 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 18:20:06.240869 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240843 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 18:20:06.240957 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.240820 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.241553 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.241534 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.241607 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.241566 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.241607 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.241586 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.242875 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.242864 2572 policy_none.go:50] "Start" Oct 31 18:20:06.242919 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.242879 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 18:20:06.242919 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.242889 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 18:20:06.244709 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.244699 2572 policy_none.go:44] "Start" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.294960 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.295743 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.296486 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.296503 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.296526 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.296566 2572 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.298877 2572 manager.go:341] "Starting Device Plugin manager" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.298915 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.298928 2572 server.go:87] "Starting device plugin registration server" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.299230 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.299242 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.299360 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.299427 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.299435 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.300113 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 18:20:06.301354 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.300157 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:06.353621 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.353590 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-47.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 18:20:06.396734 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.396661 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal"] Oct 31 18:20:06.396846 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.396736 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.397713 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.397696 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.397806 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.397731 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.397806 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.397745 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.399834 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.399819 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.399920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.399884 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.400016 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400000 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.400066 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400037 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.400583 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400568 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400593 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400574 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400614 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400617 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400624 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400629 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400596 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400649 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.400683 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.400662 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.402259 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.402238 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-47.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 18:20:06.402708 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.402689 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.402809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.402725 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.403120 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.402238 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-47.us-west-1.compute.internal.18e3b0322f54a3a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-47.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-47.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:06.212518824 +0000 UTC m=+0.370319962,LastTimestamp:2026-10-31 18:20:06.212518824 +0000 UTC m=+0.370319962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-47.us-west-1.compute.internal,}" Oct 31 18:20:06.403439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.403419 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.403535 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.403489 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.403535 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.403506 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.418679 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.418651 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.421587 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.421570 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 18:20:06.421674 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.421619 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dk4tk" Oct 31 18:20:06.425404 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.425386 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.454712 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.454684 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 18:20:06.454712 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.454684 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 31 18:20:06.454712 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.454692 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-47.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 18:20:06.454936 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.454832 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-47.us-west-1.compute.internal.18e3b03230fd448d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-47.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-47.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-47.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:06.240347277 +0000 UTC m=+0.398148415,LastTimestamp:2026-10-31 18:20:06.240347277 +0000 UTC m=+0.398148415,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-47.us-west-1.compute.internal,}" Oct 31 18:20:06.455007 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.454973 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 18:20:06.510490 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.510465 2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-47.us-west-1.compute.internal\" not found" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.524617 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.524598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d60099914d5eef84a51e7038f8e20de-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"4d60099914d5eef84a51e7038f8e20de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.524678 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.524621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.524678 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.524639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.625196 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.625166 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d60099914d5eef84a51e7038f8e20de-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"4d60099914d5eef84a51e7038f8e20de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.625321 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.625217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.625321 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.625251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.625321 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.625266 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d60099914d5eef84a51e7038f8e20de-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"4d60099914d5eef84a51e7038f8e20de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.625321 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.625285 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.625449 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.625324 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.655910 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.655865 2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-47.us-west-1.compute.internal\" not found" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.710990 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.710965 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:06.711950 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.711928 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:06.712061 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.711964 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:06.712061 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.711978 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:06.712061 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.712012 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.723260 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.723244 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.730617 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.730600 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.768931 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:06.768912 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:06.769004 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.768933 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-47.us-west-1.compute.internal\": node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:06.876483 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.876455 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:06.976816 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:06.976755 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.077745 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.077716 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.148339 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.148308 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 18:20:07.148750 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.148445 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7971b018aa4c4793a97432a2228923c-30a6b095e3cfe1a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-47.us-west-1.compute.internal.18e3b03230fd448d\": read tcp 10.0.140.47:50512->3.147.213.70:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-47.us-west-1.compute.internal.18e3b03230fd448d default 4829 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-47.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-47.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-47.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:06 +0000 UTC,LastTimestamp:2026-10-31 18:20:06.40060519 +0000 UTC m=+0.558406330,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-47.us-west-1.compute.internal,}" Oct 31 18:20:07.178611 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.178584 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.279312 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.279251 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.379701 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.379678 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.427822 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.427791 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 18:15:06 +0000 UTC" deadline="2028-04-26 04:07:13.673006293 +0000 UTC" Oct 31 18:20:07.427822 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.427816 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13017h47m6.245193412s" Oct 31 18:20:07.478809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.478787 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:20:07.479661 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.479646 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:20:07.479757 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.479681 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:20:07.479757 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.479695 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:20:07.479850 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.479755 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.561202 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.561133 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:20:07.580570 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:07.580523 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:20:07.608968 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.608950 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:20:07.622743 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.622721 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:07.680547 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.680528 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 18:20:07.682525 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.682508 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:20:07.741508 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.741481 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 18:20:07.783485 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.783463 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:20:07.884306 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:07.884262 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d60099914d5eef84a51e7038f8e20de.slice/crio-aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72 WatchSource:0}: Error finding container aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72: Status 404 returned error can't find the container with id aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72 Oct 31 18:20:07.884725 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:07.884702 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9182be3e97d0133207671d4d8f6ed1f3.slice/crio-f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475 WatchSource:0}: Error finding container f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475: Status 404 returned error can't find the container with id f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475 Oct 31 18:20:07.889288 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:07.889272 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 18:20:08.054590 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.054562 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:20:08.196305 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.196230 2572 apiserver.go:52] "Watching apiserver" Oct 31 18:20:08.222602 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.222573 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 18:20:08.302474 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.302422 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"4d60099914d5eef84a51e7038f8e20de","Type":"ContainerStarted","Data":"aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72"} Oct 31 18:20:08.303509 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.303488 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"9182be3e97d0133207671d4d8f6ed1f3","Type":"ContainerStarted","Data":"f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475"} Oct 31 18:20:08.306445 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.306427 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 18:20:08.306743 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.306727 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-bch5w","kube-system/konnectivity-agent-v2bs5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s","openshift-cluster-node-tuning-operator/tuned-sklpw","openshift-dns/node-resolver-z89h2","openshift-image-registry/node-ca-hf5q2","openshift-multus/multus-pz8h8","openshift-ovn-kubernetes/ovnkube-node-pbjfs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-xkth6","openshift-network-operator/iptables-alerter-rcrmh","kube-system/global-pull-secret-syncer-tczvl","kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-g94x2"] Oct 31 18:20:08.310719 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.310701 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.312936 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.312912 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.315338 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.315319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.317615 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.317590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.319905 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.319886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.323036 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.323012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.325573 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.325542 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.327906 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.327888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.330249 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.330230 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.334077 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.334059 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:08.334172 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.334117 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:08.336397 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-lib-modules\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.336489 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ce43312-6761-45de-8350-d347681cb539-hosts-file\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.336489 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336432 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0906409-dea7-4779-9d86-4c9bfee0ee91-host\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.336489 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336441 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:08.336489 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksmw\" (UniqueName: \"kubernetes.io/projected/d0906409-dea7-4779-9d86-4c9bfee0ee91-kube-api-access-cksmw\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.336489 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-cnibin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.336493 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-k8s-cni-cncf-io\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336612 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-hostroot\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336642 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4nj\" (UniqueName: \"kubernetes.io/projected/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-kube-api-access-xh4nj\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336743 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qp6\" (UniqueName: \"kubernetes.io/projected/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kube-api-access-n4qp6\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.336764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-system-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-device-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336848 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336869 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-systemd\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336889 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-run\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-host\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-tuned\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-tmp\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0906409-dea7-4779-9d86-4c9bfee0ee91-serviceca\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336975 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-kubelet\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.336993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-etc-kubernetes\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337097 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-modprobe-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-conf\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337139 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-sys\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zn6x\" (UniqueName: \"kubernetes.io/projected/8ce43312-6761-45de-8350-d347681cb539-kube-api-access-5zn6x\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-multus-daemon-config\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337297 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-host-slash\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337344 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-kubernetes\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-var-lib-kubelet\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-cni-binary-copy\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-netns\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337458 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbr4h\" (UniqueName: \"kubernetes.io/projected/335db604-5a37-42ad-a5b6-4536a8f5da96-kube-api-access-wbr4h\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce43312-6761-45de-8350-d347681cb539-tmp-dir\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337499 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-bin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-conf-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-multus-certs\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csg5w\" (UniqueName: \"kubernetes.io/projected/474a81eb-340b-45fe-8033-0deaecf35f03-kube-api-access-csg5w\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.337758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-iptables-alerter-script\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.338304 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337661 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.338304 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.338304 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysconfig\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.338304 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-socket-dir-parent\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.338304 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.337818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-multus\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.338760 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.338736 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.338870 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.338794 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:08.367411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.367097 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.367411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.367129 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 18:20:08.367411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.367177 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.367411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.367221 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 18:20:08.367411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.367367 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-27pxv\"" Oct 31 18:20:08.367922 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.367901 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.368127 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.368114 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4kdr\"" Oct 31 18:20:08.368239 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.368225 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.368393 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.368372 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 18:20:08.370231 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.370214 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.370231 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.370219 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 18:20:08.370494 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.370479 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgxxf\"" Oct 31 18:20:08.370562 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.370546 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.372651 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.372603 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.372759 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.372746 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.372806 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.372792 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72cf5\"" Oct 31 18:20:08.374862 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.374849 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.375002 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.374989 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.375064 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.375050 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xr757\"" Oct 31 18:20:08.378150 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.378135 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.378268 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.378245 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 18:20:08.378426 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.378414 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.415823 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.415803 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 18:20:08.415823 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.415814 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 18:20:08.416120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416105 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 18:20:08.416120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416109 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w82k4\"" Oct 31 18:20:08.416278 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416263 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 18:20:08.416326 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416294 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 18:20:08.416372 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416324 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-szhfk\"" Oct 31 18:20:08.416372 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416345 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 18:20:08.416372 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416346 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 18:20:08.416836 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.416823 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xfhz\"" Oct 31 18:20:08.420552 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.420539 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 18:20:08.420598 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.420541 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 18:20:08.420823 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.420809 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m5pkq\"" Oct 31 18:20:08.420858 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.420808 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 18:20:08.420858 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.420826 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 18:20:08.420858 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.420835 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zt98q" Oct 31 18:20:08.423700 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.423686 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 18:20:08.438676 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-run\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.438741 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-host\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.438741 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-tmp\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.438741 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd79\" (UniqueName: \"kubernetes.io/projected/89f9210d-bf91-47c5-a4ab-318f8da1985d-kube-api-access-nnd79\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-config\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-tuned\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438782 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx2g5\" (UniqueName: \"kubernetes.io/projected/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-kube-api-access-cx2g5\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-host\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438806 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6xm\" (UniqueName: \"kubernetes.io/projected/00096998-7fc2-406c-b705-c4f80a552391-kube-api-access-bp6xm\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438840 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-run\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.438887 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-kubelet\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-etc-kubernetes\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438949 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-kubelet\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.438997 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439002 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-etc-kubernetes\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-conf\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-sys\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439074 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zn6x\" (UniqueName: \"kubernetes.io/projected/8ce43312-6761-45de-8350-d347681cb539-kube-api-access-5zn6x\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-node-log\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439108 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-conf\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-host-slash\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.439185 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439166 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-sys\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439186 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-host-slash\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439210 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-kubernetes\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-var-lib-kubelet\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439243 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-kubernetes\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439276 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-var-lib-kubelet\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-konnectivity-ca\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-cni-binary-copy\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-netns\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbr4h\" (UniqueName: \"kubernetes.io/projected/335db604-5a37-42ad-a5b6-4536a8f5da96-kube-api-access-wbr4h\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439369 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-netd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-env-overrides\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439408 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-netns\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439426 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-system-cni-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-cnibin\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-iptables-alerter-script\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.439654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysconfig\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksmw\" (UniqueName: \"kubernetes.io/projected/d0906409-dea7-4779-9d86-4c9bfee0ee91-kube-api-access-cksmw\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439769 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-netns\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-socket-dir-parent\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-multus\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439901 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-cni-binary-copy\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-lib-modules\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-kubelet\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.439996 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-socket-dir-parent\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440036 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-multus\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440040 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-lib-modules\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440090 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysconfig\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-systemd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.440408 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-etc-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4nj\" (UniqueName: \"kubernetes.io/projected/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-kube-api-access-xh4nj\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qp6\" (UniqueName: \"kubernetes.io/projected/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kube-api-access-n4qp6\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-systemd\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440266 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440282 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-iptables-alerter-script\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440338 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-systemd\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440398 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-system-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440444 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-system-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0906409-dea7-4779-9d86-4c9bfee0ee91-serviceca\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440481 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-systemd-units\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440508 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-var-lib-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441138 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00096998-7fc2-406c-b705-c4f80a552391-ovn-node-metrics-cert\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440556 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-agent-certs\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-binary-copy\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-slash\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440672 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-modprobe-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-script-lib\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-multus-daemon-config\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440783 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0906409-dea7-4779-9d86-4c9bfee0ee91-serviceca\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440823 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-modprobe-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440840 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce43312-6761-45de-8350-d347681cb539-tmp-dir\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-dbus\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440908 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-ovn\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-bin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.441854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.440976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-conf-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441013 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-conf-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441047 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-multus-certs\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441057 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-bin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce43312-6761-45de-8350-d347681cb539-tmp-dir\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441079 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-multus-certs\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg5w\" (UniqueName: \"kubernetes.io/projected/474a81eb-340b-45fe-8033-0deaecf35f03-kube-api-access-csg5w\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ce43312-6761-45de-8350-d347681cb539-hosts-file\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441151 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0906409-dea7-4779-9d86-4c9bfee0ee91-host\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ce43312-6761-45de-8350-d347681cb539-hosts-file\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-cnibin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441254 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-multus-daemon-config\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441277 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0906409-dea7-4779-9d86-4c9bfee0ee91-host\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441305 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-cnibin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-k8s-cni-cncf-io\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441327 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-k8s-cni-cncf-io\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-hostroot\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441369 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-kubelet-config\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441389 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-hostroot\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-log-socket\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-bin\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-device-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441584 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441603 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441610 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-device-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.441705 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.442500 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-tuned\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.442920 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.442539 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-tmp\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.502056 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.501990 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksmw\" (UniqueName: \"kubernetes.io/projected/d0906409-dea7-4779-9d86-4c9bfee0ee91-kube-api-access-cksmw\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.502274 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.502252 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qp6\" (UniqueName: \"kubernetes.io/projected/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kube-api-access-n4qp6\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.503326 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.503305 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbr4h\" (UniqueName: \"kubernetes.io/projected/335db604-5a37-42ad-a5b6-4536a8f5da96-kube-api-access-wbr4h\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.503326 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.503318 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zn6x\" (UniqueName: \"kubernetes.io/projected/8ce43312-6761-45de-8350-d347681cb539-kube-api-access-5zn6x\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.503460 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.503319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4nj\" (UniqueName: \"kubernetes.io/projected/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-kube-api-access-xh4nj\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.503460 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.503391 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg5w\" (UniqueName: \"kubernetes.io/projected/474a81eb-340b-45fe-8033-0deaecf35f03-kube-api-access-csg5w\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.542932 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.542907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:08.543008 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.542936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-kubelet-config\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.543008 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.542951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-log-socket\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543008 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.542967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543008 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.542983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-bin\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543030 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-bin\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-kubelet-config\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.543175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543060 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd79\" (UniqueName: \"kubernetes.io/projected/89f9210d-bf91-47c5-a4ab-318f8da1985d-kube-api-access-nnd79\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.543175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-log-socket\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-config\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx2g5\" (UniqueName: \"kubernetes.io/projected/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-kube-api-access-cx2g5\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6xm\" (UniqueName: \"kubernetes.io/projected/00096998-7fc2-406c-b705-c4f80a552391-kube-api-access-bp6xm\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-node-log\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-konnectivity-ca\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543322 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-node-log\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-netd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543409 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-env-overrides\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-netd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543449 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-system-cni-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.543492 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-cnibin\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543517 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-system-cni-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-netns\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543581 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-cnibin\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543610 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-netns\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-kubelet\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543646 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-kubelet\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543645 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-systemd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543672 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-etc-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543680 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-systemd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.543610 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543747 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-etc-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.543757 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:09.043731756 +0000 UTC m=+3.201532906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.543774 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:08.544030 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543775 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-config\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543784 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.543818 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:09.043803782 +0000 UTC m=+3.201604926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543826 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-env-overrides\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543855 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-systemd-units\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-var-lib-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00096998-7fc2-406c-b705-c4f80a552391-ovn-node-metrics-cert\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543908 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-systemd-units\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-agent-certs\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543946 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-konnectivity-ca\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543955 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-binary-copy\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543959 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-var-lib-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.543988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-slash\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-script-lib\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545044 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544085 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-slash\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-dbus\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-ovn\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544141 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-dbus\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544160 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544215 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544225 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-ovn\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544392 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544414 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544432 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-binary-copy\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.545861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.544626 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-script-lib\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.546356 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.546026 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00096998-7fc2-406c-b705-c4f80a552391-ovn-node-metrics-cert\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.546772 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.546754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-agent-certs\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.601373 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.601351 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 18:20:08.601373 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.601373 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 18:20:08.601515 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.601383 2572 projected.go:196] Error preparing data for projected volume kube-api-access-4xpmx for pod openshift-network-diagnostics/network-check-target-bch5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:08.601515 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:08.601435 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx podName:adfe1fd5-4835-4b2b-87d6-d218cfcd6b85 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:09.101421858 +0000 UTC m=+3.259223001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4xpmx" (UniqueName: "kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx") pod "network-check-target-bch5w" (UID: "adfe1fd5-4835-4b2b-87d6-d218cfcd6b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:08.603646 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.603627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx2g5\" (UniqueName: \"kubernetes.io/projected/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-kube-api-access-cx2g5\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:08.603876 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.603859 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6xm\" (UniqueName: \"kubernetes.io/projected/00096998-7fc2-406c-b705-c4f80a552391-kube-api-access-bp6xm\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.604225 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.604191 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd79\" (UniqueName: \"kubernetes.io/projected/89f9210d-bf91-47c5-a4ab-318f8da1985d-kube-api-access-nnd79\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.622980 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.622959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pz8h8" Oct 31 18:20:08.629401 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.629384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:20:08.633136 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.633116 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a81eb_340b_45fe_8033_0deaecf35f03.slice/crio-995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d WatchSource:0}: Error finding container 995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d: Status 404 returned error can't find the container with id 995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d Oct 31 18:20:08.637783 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.637757 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce9a5a8_8119_4fd2_baec_f2cb3b0d1759.slice/crio-5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766 WatchSource:0}: Error finding container 5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766: Status 404 returned error can't find the container with id 5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766 Oct 31 18:20:08.644154 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.644138 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:20:08.644280 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.644239 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:20:08.649076 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.649062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z89h2" Oct 31 18:20:08.654208 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.654160 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512c0bfc_c41f_4abd_b4cd_c85d358fac3e.slice/crio-3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf WatchSource:0}: Error finding container 3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf: Status 404 returned error can't find the container with id 3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf Oct 31 18:20:08.654659 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.654630 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:20:08.657862 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.657840 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335db604_5a37_42ad_a5b6_4536a8f5da96.slice/crio-2b3cf2ee174c978e0c08f76917736a199af59508507d47c8917718072a1f77f5 WatchSource:0}: Error finding container 2b3cf2ee174c978e0c08f76917736a199af59508507d47c8917718072a1f77f5: Status 404 returned error can't find the container with id 2b3cf2ee174c978e0c08f76917736a199af59508507d47c8917718072a1f77f5 Oct 31 18:20:08.661534 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.661488 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce43312_6761_45de_8350_d347681cb539.slice/crio-1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0 WatchSource:0}: Error finding container 1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0: Status 404 returned error can't find the container with id 1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0 Oct 31 18:20:08.664507 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.664472 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0906409_dea7_4779_9d86_4c9bfee0ee91.slice/crio-1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943 WatchSource:0}: Error finding container 1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943: Status 404 returned error can't find the container with id 1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943 Oct 31 18:20:08.674120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.674093 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:08.683224 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.683182 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:20:08.684419 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.684395 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc74a8fe_8053_4b78_b8bf_569e5e34a89f.slice/crio-7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69 WatchSource:0}: Error finding container 7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69: Status 404 returned error can't find the container with id 7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69 Oct 31 18:20:08.691232 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.691212 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f9210d_bf91_47c5_a4ab_318f8da1985d.slice/crio-1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf WatchSource:0}: Error finding container 1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf: Status 404 returned error can't find the container with id 1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf Oct 31 18:20:08.694317 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:08.694290 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:08.704516 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:08.704493 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00096998_7fc2_406c_b705_c4f80a552391.slice/crio-0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a WatchSource:0}: Error finding container 0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a: Status 404 returned error can't find the container with id 0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a Oct 31 18:20:09.048333 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.048297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:09.048532 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.048353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:09.048532 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.048474 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:09.048532 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.048528 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:10.04850924 +0000 UTC m=+4.206310387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:09.048717 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.048580 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:09.048717 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.048604 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:10.048596664 +0000 UTC m=+4.206397791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:09.149560 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.149523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:09.149708 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.149693 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 18:20:09.149758 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.149716 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 18:20:09.149758 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.149726 2572 projected.go:196] Error preparing data for projected volume kube-api-access-4xpmx for pod openshift-network-diagnostics/network-check-target-bch5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:09.149816 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:09.149776 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx podName:adfe1fd5-4835-4b2b-87d6-d218cfcd6b85 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:10.149761787 +0000 UTC m=+4.307562929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xpmx" (UniqueName: "kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx") pod "network-check-target-bch5w" (UID: "adfe1fd5-4835-4b2b-87d6-d218cfcd6b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:09.307382 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.307281 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sklpw" event={"ID":"335db604-5a37-42ad-a5b6-4536a8f5da96","Type":"ContainerStarted","Data":"2b3cf2ee174c978e0c08f76917736a199af59508507d47c8917718072a1f77f5"} Oct 31 18:20:09.308890 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.308854 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pz8h8" event={"ID":"474a81eb-340b-45fe-8033-0deaecf35f03","Type":"ContainerStarted","Data":"995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d"} Oct 31 18:20:09.310621 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.310511 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a"} Oct 31 18:20:09.311796 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.311775 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerStarted","Data":"1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf"} Oct 31 18:20:09.313002 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.312972 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2bs5" event={"ID":"cc74a8fe-8053-4b78-b8bf-569e5e34a89f","Type":"ContainerStarted","Data":"7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69"} Oct 31 18:20:09.314221 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.314194 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hf5q2" event={"ID":"d0906409-dea7-4779-9d86-4c9bfee0ee91","Type":"ContainerStarted","Data":"1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943"} Oct 31 18:20:09.315409 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.315382 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z89h2" event={"ID":"8ce43312-6761-45de-8350-d347681cb539","Type":"ContainerStarted","Data":"1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0"} Oct 31 18:20:09.316889 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.316864 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" event={"ID":"512c0bfc-c41f-4abd-b4cd-c85d358fac3e","Type":"ContainerStarted","Data":"3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf"} Oct 31 18:20:09.318165 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.318140 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rcrmh" event={"ID":"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759","Type":"ContainerStarted","Data":"5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766"} Oct 31 18:20:09.423732 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.423674 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:08 +0000 UTC" deadline="2028-04-05 02:00:24.130043812 +0000 UTC" Oct 31 18:20:09.423732 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:09.423728 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12511h40m14.706319409s" Oct 31 18:20:10.064503 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.064322 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:10.064503 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.064378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:10.064711 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.064579 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:10.064711 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.064638 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:12.06461786 +0000 UTC m=+6.222419003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:10.064711 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.064519 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:10.064711 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.064685 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:12.064674933 +0000 UTC m=+6.222476079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:10.165645 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.165595 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:10.165881 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.165859 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 18:20:10.165961 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.165888 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 18:20:10.165961 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.165932 2572 projected.go:196] Error preparing data for projected volume kube-api-access-4xpmx for pod openshift-network-diagnostics/network-check-target-bch5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:10.166071 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.165992 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx podName:adfe1fd5-4835-4b2b-87d6-d218cfcd6b85 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:12.165971064 +0000 UTC m=+6.323772195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xpmx" (UniqueName: "kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx") pod "network-check-target-bch5w" (UID: "adfe1fd5-4835-4b2b-87d6-d218cfcd6b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:10.300205 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.299656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:10.300205 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.299785 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:10.300205 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.299879 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:10.300205 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.299966 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:10.300205 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.300026 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:10.300205 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:10.300098 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:10.424352 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.424294 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:08 +0000 UTC" deadline="2028-06-22 07:31:31.069889815 +0000 UTC" Oct 31 18:20:10.424352 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:10.424335 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14389h11m20.645558786s" Oct 31 18:20:12.081099 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.080586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:12.081099 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.080763 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:12.081099 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.081003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:12.081099 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.081009 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:16.080984719 +0000 UTC m=+10.238785864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:12.081099 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.081027 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:12.081099 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.081065 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:16.081053957 +0000 UTC m=+10.238855102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:12.182385 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.182350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:12.182576 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.182559 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 18:20:12.182621 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.182583 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 18:20:12.182621 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.182597 2572 projected.go:196] Error preparing data for projected volume kube-api-access-4xpmx for pod openshift-network-diagnostics/network-check-target-bch5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:12.182676 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.182663 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx podName:adfe1fd5-4835-4b2b-87d6-d218cfcd6b85 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:16.182642152 +0000 UTC m=+10.340443297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xpmx" (UniqueName: "kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx") pod "network-check-target-bch5w" (UID: "adfe1fd5-4835-4b2b-87d6-d218cfcd6b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:12.297426 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.297395 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:12.297571 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.297544 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:12.297650 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.297629 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:12.302443 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.302407 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:12.303324 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.302620 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:12.303324 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:12.302780 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:12.334731 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.334697 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerStarted","Data":"ca14fc0f427d617e21e1596315bdb3b21c92086bca8338a1cc46c596104de607"} Oct 31 18:20:12.339727 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.339695 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"4d60099914d5eef84a51e7038f8e20de","Type":"ContainerStarted","Data":"0d5fb4755a93f8fddc12667b5ac35cae974d8698f4e55152dae9d463d82a5bc4"} Oct 31 18:20:12.345700 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.345672 2572 generic.go:356] "Generic (PLEG): container finished" podID="9182be3e97d0133207671d4d8f6ed1f3" containerID="0b51e91abd07adbc6805190e7bc2ce8a4d6567ec4920d250d01deb5fd4d68cf4" exitCode=0 Oct 31 18:20:12.345814 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.345713 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"9182be3e97d0133207671d4d8f6ed1f3","Type":"ContainerDied","Data":"0b51e91abd07adbc6805190e7bc2ce8a4d6567ec4920d250d01deb5fd4d68cf4"} Oct 31 18:20:12.764804 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:12.764435 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" podStartSLOduration=4.764415346 podStartE2EDuration="4.764415346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:12.642102902 +0000 UTC m=+6.799904058" watchObservedRunningTime="2026-10-31 18:20:12.764415346 +0000 UTC m=+6.922216501" Oct 31 18:20:13.350928 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:13.350875 2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="ca14fc0f427d617e21e1596315bdb3b21c92086bca8338a1cc46c596104de607" exitCode=0 Oct 31 18:20:13.351454 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:13.350978 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerDied","Data":"ca14fc0f427d617e21e1596315bdb3b21c92086bca8338a1cc46c596104de607"} Oct 31 18:20:13.363815 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:13.363782 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"9182be3e97d0133207671d4d8f6ed1f3","Type":"ContainerStarted","Data":"62fec414e7240bf11b8f4ab16935a48dee9792e0b9176dfb7508651c6c765d4e"} Oct 31 18:20:14.299981 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:14.299424 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:14.299981 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:14.299560 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:14.299981 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:14.299637 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:14.299981 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:14.299728 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:14.299981 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:14.299796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:14.299981 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:14.299874 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:14.369430 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:14.369361 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2bs5" event={"ID":"cc74a8fe-8053-4b78-b8bf-569e5e34a89f","Type":"ContainerStarted","Data":"5251f17b4a3af53ac5dbe7b6f0cd17e06c06776ac63014f7ca3ac24e21823973"} Oct 31 18:20:14.489470 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:14.489408 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" podStartSLOduration=6.489386401 podStartE2EDuration="6.489386401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:13.600761045 +0000 UTC m=+7.758562198" watchObservedRunningTime="2026-10-31 18:20:14.489386401 +0000 UTC m=+8.647187555" Oct 31 18:20:15.376786 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:15.376696 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hf5q2" event={"ID":"d0906409-dea7-4779-9d86-4c9bfee0ee91","Type":"ContainerStarted","Data":"797986491cef159677d63b4156dcd48ee34aad5c9ae68fdf45bf7089d69310f5"} Oct 31 18:20:15.496614 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:15.496006 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v2bs5" podStartSLOduration=3.633099194 podStartE2EDuration="8.495987221s" totalImagesPullingTime="4.862888027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:14.490236276 +0000 UTC m=+8.648037423" watchObservedRunningTime="2026-10-31 18:20:15.495987221 +0000 UTC m=+9.653788378" Oct 31 18:20:15.496614 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:15.496302 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hf5q2" podStartSLOduration=2.759849434 podStartE2EDuration="8.496289567s" totalImagesPullingTime="5.736440133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:15.49597856 +0000 UTC m=+9.653779712" watchObservedRunningTime="2026-10-31 18:20:15.496289567 +0000 UTC m=+9.654090737" Oct 31 18:20:16.118363 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.118293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:16.118363 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.118335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:16.118521 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.118448 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:16.118521 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.118492 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:24.118480708 +0000 UTC m=+18.276281847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:16.118594 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.118543 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:16.118594 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.118567 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:24.118560101 +0000 UTC m=+18.276361240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:16.218792 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.218736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:16.218948 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.218933 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 18:20:16.219017 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.218955 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 18:20:16.219017 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.218968 2572 projected.go:196] Error preparing data for projected volume kube-api-access-4xpmx for pod openshift-network-diagnostics/network-check-target-bch5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:16.219127 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.219019 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx podName:adfe1fd5-4835-4b2b-87d6-d218cfcd6b85 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:24.219001959 +0000 UTC m=+18.376803100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xpmx" (UniqueName: "kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx") pod "network-check-target-bch5w" (UID: "adfe1fd5-4835-4b2b-87d6-d218cfcd6b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:16.297357 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.297315 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:16.297357 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.297361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:16.297523 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.297459 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:16.297582 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.297546 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:16.297617 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.297602 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:16.297701 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:16.297683 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:16.382053 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:16.381979 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" event={"ID":"512c0bfc-c41f-4abd-b4cd-c85d358fac3e","Type":"ContainerStarted","Data":"551f7211125ca7dd0b5446ec220bdf60e4eac9e62d610344006c720507ab662d"} Oct 31 18:20:17.385490 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:17.385389 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z89h2" event={"ID":"8ce43312-6761-45de-8350-d347681cb539","Type":"ContainerStarted","Data":"e14acb3346df16b5d4bc3bd51e5c97add2cf791f77cf606198d3defc15498068"} Oct 31 18:20:17.496828 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:17.496774 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-z89h2" podStartSLOduration=2.0944201 podStartE2EDuration="10.496754026s" totalImagesPullingTime="8.402333926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:17.495690968 +0000 UTC m=+11.653492117" watchObservedRunningTime="2026-10-31 18:20:17.496754026 +0000 UTC m=+11.654555176" Oct 31 18:20:17.560008 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:17.559971 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 18:20:17.747912 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:17.747636 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:17.748053 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:17.748006 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:18.297714 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.297489 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:18.297714 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.297515 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:18.297714 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:18.297656 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:18.297714 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.297689 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:18.297994 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:18.297773 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:18.297994 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:18.297837 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:18.306396 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.306311 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T18:20:17.559988013Z","UUID":"560da89e-980c-4e43-a848-5bf6f3187af7","Handler":null,"Name":"","Endpoint":""} Oct 31 18:20:18.308442 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.308390 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 18:20:18.308442 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.308411 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 18:20:18.390274 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.390166 2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="441d1d525a441b28f251fd2fd64267261bc75c0e6d8c87b1e353b2d33645a194" exitCode=0 Oct 31 18:20:18.390274 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.390230 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerDied","Data":"441d1d525a441b28f251fd2fd64267261bc75c0e6d8c87b1e353b2d33645a194"} Oct 31 18:20:18.393039 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.393005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" event={"ID":"512c0bfc-c41f-4abd-b4cd-c85d358fac3e","Type":"ContainerStarted","Data":"cfcca0c4ea71670c7d48433fa2becfe3c33e6d902ad7a818def2434478792765"} Oct 31 18:20:18.394807 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.394439 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pz8h8" event={"ID":"474a81eb-340b-45fe-8033-0deaecf35f03","Type":"ContainerStarted","Data":"f65c1ec01fcfaa89e9369f759b2c1f09cbfddd2e904e20cd243ebd3b7e31cca2"} Oct 31 18:20:18.394807 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.394464 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 18:20:18.617910 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:18.617860 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pz8h8" podStartSLOduration=2.840796906 podStartE2EDuration="11.617840863s" totalImagesPullingTime="8.777043957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:18.617650988 +0000 UTC m=+12.775452137" watchObservedRunningTime="2026-10-31 18:20:18.617840863 +0000 UTC m=+12.775642014" Oct 31 18:20:19.399108 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:19.399034 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" event={"ID":"512c0bfc-c41f-4abd-b4cd-c85d358fac3e","Type":"ContainerStarted","Data":"a675274d7fe02fc6f9bbd8ddebb5efd47641da1f389e1dd8e418d880c9d33f8f"} Oct 31 18:20:19.400784 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:19.400761 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rcrmh" event={"ID":"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759","Type":"ContainerStarted","Data":"15199e525ec601e409233e953b14528e1026b58b7c70fceca7835e470dc6e250"} Oct 31 18:20:19.518278 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:19.518222 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" podStartSLOduration=2.476130529 podStartE2EDuration="12.518206674s" totalImagesPullingTime="10.042076145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:19.517514504 +0000 UTC m=+13.675315652" watchObservedRunningTime="2026-10-31 18:20:19.518206674 +0000 UTC m=+13.676007820" Oct 31 18:20:20.297502 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.297473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:20.297667 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.297536 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:20.297667 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.297545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:20.297667 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:20.297633 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:20.297832 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:20.297739 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:20.297882 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:20.297834 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:20.404535 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.404499 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sklpw" event={"ID":"335db604-5a37-42ad-a5b6-4536a8f5da96","Type":"ContainerStarted","Data":"0ae3213a2358f64b8061cfb695bfa620919372bcb9498f376b0741d10df9849b"} Oct 31 18:20:20.407589 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.407526 2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="6c8a554dbccad95135c3fa9a5ce9ac0c2eceb6f860495e4026413980faf3f935" exitCode=0 Oct 31 18:20:20.407725 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.407616 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerDied","Data":"6c8a554dbccad95135c3fa9a5ce9ac0c2eceb6f860495e4026413980faf3f935"} Oct 31 18:20:20.518017 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.517965 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rcrmh" podStartSLOduration=4.734169794 podStartE2EDuration="13.517945804s" totalImagesPullingTime="8.78377601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:19.629552809 +0000 UTC m=+13.787353958" watchObservedRunningTime="2026-10-31 18:20:20.517945804 +0000 UTC m=+14.675746951" Oct 31 18:20:20.518384 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:20.518357 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sklpw" podStartSLOduration=2.738255021 podStartE2EDuration="13.518350373s" totalImagesPullingTime="10.780095352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:20.518088494 +0000 UTC m=+14.675889623" watchObservedRunningTime="2026-10-31 18:20:20.518350373 +0000 UTC m=+14.676151520" Oct 31 18:20:22.302546 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:22.302352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:22.303110 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:22.302386 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:22.303110 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:22.302630 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:22.303110 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:22.302722 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:22.303110 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:22.302421 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:22.303110 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:22.302810 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:22.414256 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:22.414224 2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="ad5e1f336d83c1502f583a80d8a2d167cce2068193a0c4c22e3025026883c845" exitCode=0 Oct 31 18:20:22.414429 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:22.414277 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerDied","Data":"ad5e1f336d83c1502f583a80d8a2d167cce2068193a0c4c22e3025026883c845"} Oct 31 18:20:24.185471 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:24.185428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:24.185916 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:24.185494 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:24.185916 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.185621 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:24.185916 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.185681 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.185664206 +0000 UTC m=+34.343465332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:24.185916 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.185764 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:24.185916 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.185798 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.185787498 +0000 UTC m=+34.343588650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:24.286804 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:24.286762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:24.286961 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.286914 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 18:20:24.286961 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.286933 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 18:20:24.286961 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.286945 2572 projected.go:196] Error preparing data for projected volume kube-api-access-4xpmx for pod openshift-network-diagnostics/network-check-target-bch5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:24.287060 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.286996 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx podName:adfe1fd5-4835-4b2b-87d6-d218cfcd6b85 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.286983712 +0000 UTC m=+34.444784851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xpmx" (UniqueName: "kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx") pod "network-check-target-bch5w" (UID: "adfe1fd5-4835-4b2b-87d6-d218cfcd6b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 18:20:24.302220 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:24.301983 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:24.302220 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:24.302081 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:24.302220 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.302092 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:24.302220 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:24.302113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:24.302220 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.302199 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:24.302508 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:24.302324 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:25.424839 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.424602 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"d927c512242f6721e1a0abe08ee8beb1b3b65e4ac1cbcdd0e4fd9441d213509a"} Oct 31 18:20:25.425166 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.424941 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"119aaf25e47f7e25e25c96c52948f6fb57e6dd15b811a06b446e0047fe28e23a"} Oct 31 18:20:25.425166 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.425006 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"11bcfeb2e61906abb44a911140152eb14aa5923a6509c281109f81588be1d9dd"} Oct 31 18:20:25.425166 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.425055 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"8ffaaf6eeb04fc3286a773b75aa15268109c965bb78f9daa39167ecad256d680"} Oct 31 18:20:25.425166 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.425107 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"d4076bfc67dcf51c989ca0633d891685eb933616b5422f4ddb7835f9d7265fdb"} Oct 31 18:20:25.429550 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.429526 2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="16e3463c79cb2e4d1c833024ebb8f21c801676d2fc907972a9034f74b42a820b" exitCode=0 Oct 31 18:20:25.429647 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:25.429562 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerDied","Data":"16e3463c79cb2e4d1c833024ebb8f21c801676d2fc907972a9034f74b42a820b"} Oct 31 18:20:26.299379 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:26.299355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:26.299537 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:26.299463 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:26.299537 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:26.299465 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:26.299618 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:26.299544 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:26.299618 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:26.299563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:26.299687 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:26.299635 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:26.434252 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:26.434209 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"b79c4d0f4a61c29362bb07a55a6c1227615afa87a220487508da9777771c2ea7"} Oct 31 18:20:26.437067 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:26.437043 2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="9a17508a36c2935a03ec7d802abb7b202d3ea6d2f9ab640c18cc88658b8673f6" exitCode=0 Oct 31 18:20:26.437176 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:26.437086 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerDied","Data":"9a17508a36c2935a03ec7d802abb7b202d3ea6d2f9ab640c18cc88658b8673f6"} Oct 31 18:20:27.443719 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:27.443678 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g94x2" event={"ID":"89f9210d-bf91-47c5-a4ab-318f8da1985d","Type":"ContainerStarted","Data":"7d010173076911e0b701fd23b504299a42b7b718f7223e77a96708a2a836de12"} Oct 31 18:20:27.658937 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:27.658890 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g94x2" podStartSLOduration=7.801014713 podStartE2EDuration="20.658877235s" totalImagesPullingTime="12.857862522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:27.658336441 +0000 UTC m=+21.816137590" watchObservedRunningTime="2026-10-31 18:20:27.658877235 +0000 UTC m=+21.816678383" Oct 31 18:20:28.296934 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:28.296901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:28.297082 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:28.296962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:28.297082 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:28.296965 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:28.297082 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:28.297043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:28.297252 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:28.297110 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:28.297284 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:28.297246 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:28.449460 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:28.449420 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"e88bc115213978dec3ff5605b936b68e1a1c2b3f867f57c469302c7fa0c48a07"} Oct 31 18:20:30.297177 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.297143 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:30.297788 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.297158 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:30.297788 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.297276 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:30.297788 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:30.297265 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:30.297788 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:30.297359 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:30.297788 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:30.297440 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:30.456483 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.456453 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" event={"ID":"00096998-7fc2-406c-b705-c4f80a552391","Type":"ContainerStarted","Data":"f20c42eecf72afac2e2c2d177fb3fe661426c535e0a48287c05591a7a8c04333"} Oct 31 18:20:30.456680 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.456664 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:30.456759 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.456723 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:30.471816 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.471796 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:30.724096 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:30.724015 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" podStartSLOduration=7.295769378 podStartE2EDuration="23.724003556s" totalImagesPullingTime="16.428234178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:30.723846327 +0000 UTC m=+24.881647476" watchObservedRunningTime="2026-10-31 18:20:30.724003556 +0000 UTC m=+24.881804703" Oct 31 18:20:31.458723 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:31.458697 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:31.474466 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:31.474440 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:20:32.297659 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.297627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:32.297793 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.297627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:32.298121 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:32.298093 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:32.298292 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:32.298215 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:32.298370 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.298258 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:32.298818 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:32.298790 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:32.333220 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.333197 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tczvl"] Oct 31 18:20:32.341845 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.341822 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkth6"] Oct 31 18:20:32.343588 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.343572 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bch5w"] Oct 31 18:20:32.460448 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.460422 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:32.460448 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.460430 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:32.460825 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:32.460519 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:32.460895 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:32.460872 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:32.460985 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:32.460966 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:32.461057 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:32.461041 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:33.891319 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:33.891021 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:33.891668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:33.891371 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 18:20:33.891668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:33.891619 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:20:34.297002 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:34.296975 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:34.297002 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:34.296988 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:34.297262 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:34.296991 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:34.297262 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:34.297098 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:34.297365 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:34.297248 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:34.297411 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:34.297353 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:36.301866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:36.301784 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:36.302321 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:36.301870 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:36.302321 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:36.301917 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:36.302321 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:36.301908 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:36.302321 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:36.301932 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:36.302321 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:36.302025 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:38.296792 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.296726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:38.297124 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.296731 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:38.297124 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:38.296827 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bch5w" podUID="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" Oct 31 18:20:38.297124 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:38.296896 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkth6" podUID="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" Oct 31 18:20:38.297124 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.296932 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:38.297124 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:38.296990 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tczvl" podUID="96e03cbb-2909-4802-accd-002dd2922c42" Oct 31 18:20:38.534599 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.534572 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeReady" Oct 31 18:20:38.534745 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.534678 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 18:20:38.646336 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.646255 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgpld"] Oct 31 18:20:38.707978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.707943 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v4vbz"] Oct 31 18:20:38.708160 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.708034 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:20:38.720892 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.720868 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ksnkf"] Oct 31 18:20:38.721093 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.720986 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:20:38.733347 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.733321 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8558f787fd-29dwp"] Oct 31 18:20:38.733500 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.733465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:20:38.749798 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.749777 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz"] Oct 31 18:20:38.749910 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.749880 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.760981 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.760962 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8z6s5"] Oct 31 18:20:38.761086 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.761070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:38.763324 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.763305 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.763425 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.763410 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.777479 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.777453 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59bdcd5559-qvgdn"] Oct 31 18:20:38.777911 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.777742 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.792572 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.792502 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c"] Oct 31 18:20:38.792572 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.792566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:38.802133 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.802113 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 18:20:38.802241 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.802136 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 18:20:38.802241 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.802167 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 18:20:38.802241 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.802216 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sqcvs\"" Oct 31 18:20:38.806778 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.806761 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v"] Oct 31 18:20:38.806945 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.806866 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:20:38.812309 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.812287 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 18:20:38.812480 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.812466 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6pq8w\"" Oct 31 18:20:38.822653 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.822629 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm"] Oct 31 18:20:38.823941 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.822763 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:38.838394 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.838378 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zqcnd"] Oct 31 18:20:38.838497 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.838482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:38.850833 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.850814 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t"] Oct 31 18:20:38.850946 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.850931 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:38.865841 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.865826 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xncwq"] Oct 31 18:20:38.865935 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.865923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:38.867938 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.867920 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 18:20:38.867938 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.867924 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.868137 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868115 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.868213 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868137 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.868301 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868286 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.868412 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868397 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.868468 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868412 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.868468 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868446 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 18:20:38.868468 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868452 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.868468 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868463 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.868982 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.868964 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 18:20:38.880609 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.880593 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz"] Oct 31 18:20:38.880689 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.880674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:38.890553 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.890533 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85cbfd9d9f-q2vwz"] Oct 31 18:20:38.890648 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.890636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:20:38.898575 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.898575 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrkf\" (UniqueName: \"kubernetes.io/projected/875a3d19-4e6c-4bb6-87b9-e801cd4f6604-kube-api-access-ggrkf\") pod \"kas-connection-checker-685895c7c9-xgpld\" (UID: \"875a3d19-4e6c-4bb6-87b9-e801cd4f6604\") " pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:20:38.898680 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.898680 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:38.898680 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-snapshots\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.898680 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.898792 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898712 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-service-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.898792 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wjtb\" (UniqueName: \"kubernetes.io/projected/4334dda9-cc0b-49bd-a5ca-48457df595fe-kube-api-access-6wjtb\") pod \"kas-connection-checker-685895c7c9-ksnkf\" (UID: \"4334dda9-cc0b-49bd-a5ca-48457df595fe\") " pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:20:38.898851 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:38.898851 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898812 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclxl\" (UniqueName: \"kubernetes.io/projected/b6d33e27-7f70-4b3c-b90d-927c598a0630-kube-api-access-sclxl\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.898906 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tww9t\" (UniqueName: \"kubernetes.io/projected/25825c7b-a2c6-4263-bc62-d2c85c00ec47-kube-api-access-tww9t\") pod \"kas-connection-checker-685895c7c9-v4vbz\" (UID: \"25825c7b-a2c6-4263-bc62-d2c85c00ec47\") " pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:20:38.898906 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-tmp\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.898960 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrpt\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.898992 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.899020 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.898999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:38.899052 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d33e27-7f70-4b3c-b90d-927c598a0630-serving-cert\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:38.899082 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.899082 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-default-certificate\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:38.899136 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9qh\" (UniqueName: \"kubernetes.io/projected/f17f5606-10aa-4f02-a2c4-16c28aba8540-kube-api-access-9z9qh\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:38.899136 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgj2q\" (UniqueName: \"kubernetes.io/projected/52aa29ef-34be-481f-b3a6-d78bfeb0b181-kube-api-access-dgj2q\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:38.899213 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.899251 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899221 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52aa29ef-34be-481f-b3a6-d78bfeb0b181-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:38.899251 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.899307 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:38.899307 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.899269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-stats-auth\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:38.901065 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.901050 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-px79l"] Oct 31 18:20:38.901167 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.901151 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:38.909251 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.909233 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pcvwd"] Oct 31 18:20:38.909359 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.909344 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:38.919242 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919224 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.919335 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919265 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 18:20:38.919335 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919291 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.919335 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919271 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.919335 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919326 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 18:20:38.919517 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919356 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 18:20:38.919517 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919365 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 18:20:38.919517 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919448 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 18:20:38.919517 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919466 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 18:20:38.919517 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919495 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 18:20:38.919734 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919648 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 18:20:38.919734 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919680 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.919734 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919693 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6fctz\"" Oct 31 18:20:38.919734 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919708 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-52br2\"" Oct 31 18:20:38.919904 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919739 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 18:20:38.919904 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919749 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.919904 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919791 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9kc2n\"" Oct 31 18:20:38.919904 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919793 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 18:20:38.919904 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919848 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zxchx\"" Oct 31 18:20:38.919904 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919877 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 18:20:38.920075 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.919969 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-whbd6\"" Oct 31 18:20:38.920075 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.920033 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 18:20:38.920075 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.920056 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.920160 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.920082 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6wm76\"" Oct 31 18:20:38.921541 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.921527 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 18:20:38.921630 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.921612 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7pnqt\"" Oct 31 18:20:38.921671 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.921641 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 18:20:38.922558 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922544 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgpld"] Oct 31 18:20:38.922594 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922569 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v4vbz"] Oct 31 18:20:38.922594 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922582 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ksnkf"] Oct 31 18:20:38.922594 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922594 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz"] Oct 31 18:20:38.922668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922611 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8558f787fd-29dwp"] Oct 31 18:20:38.922668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922627 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v"] Oct 31 18:20:38.922668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922640 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm"] Oct 31 18:20:38.922668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922647 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:38.922777 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922652 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8z6s5"] Oct 31 18:20:38.922803 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922777 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz"] Oct 31 18:20:38.922831 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922805 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xncwq"] Oct 31 18:20:38.922861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922830 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85cbfd9d9f-q2vwz"] Oct 31 18:20:38.922861 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922853 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59bdcd5559-qvgdn"] Oct 31 18:20:38.922913 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922867 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zqcnd"] Oct 31 18:20:38.922913 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922880 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c"] Oct 31 18:20:38.922913 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922892 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcvwd"] Oct 31 18:20:38.922987 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922915 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-px79l"] Oct 31 18:20:38.922987 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.922928 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t"] Oct 31 18:20:38.966825 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.966806 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.966825 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.966814 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 18:20:38.966958 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.966833 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 18:20:38.966958 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.966867 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.967582 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.967549 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 18:20:38.968136 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968103 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p4b7h\"" Oct 31 18:20:38.968241 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968204 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms9mk\"" Oct 31 18:20:38.968241 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968230 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.968346 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968252 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 18:20:38.968346 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968260 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.968439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968394 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pk99b\"" Oct 31 18:20:38.968439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968424 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 18:20:38.968597 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.968581 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 18:20:38.973603 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.973585 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 18:20:38.973732 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.973714 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 18:20:38.973894 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:38.973881 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5t2xq\"" Oct 31 18:20:39.000134 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-service-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.000134 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.000367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000149 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclxl\" (UniqueName: \"kubernetes.io/projected/b6d33e27-7f70-4b3c-b90d-927c598a0630-kube-api-access-sclxl\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.000367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000164 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrkf\" (UniqueName: \"kubernetes.io/projected/875a3d19-4e6c-4bb6-87b9-e801cd4f6604-kube-api-access-ggrkf\") pod \"kas-connection-checker-685895c7c9-xgpld\" (UID: \"875a3d19-4e6c-4bb6-87b9-e801cd4f6604\") " pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:20:39.000367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.000367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f957aba3-d3ed-48a0-aff7-2c01130e95ee-config\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.000367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000314 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.000614 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-tmp\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.000614 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.000512 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 18:20:39.000614 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.000614 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000596 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f957aba3-d3ed-48a0-aff7-2c01130e95ee-serving-cert\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.000809 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.000631 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.500609507 +0000 UTC m=+33.658410661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : secret "router-metrics-certs-default" not found Oct 31 18:20:39.000809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrpt\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.000809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000707 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.000809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000734 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.000809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000771 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-tmp\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.000809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8dt\" (UniqueName: \"kubernetes.io/projected/f58038f2-47ce-43fc-bfa5-116eed7f4776-kube-api-access-lj8dt\") pod \"volume-data-source-validator-58c9f5f46d-98g5c\" (UID: \"f58038f2-47ce-43fc-bfa5-116eed7f4776\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:20:39.001003 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xc4q\" (UniqueName: \"kubernetes.io/projected/f957aba3-d3ed-48a0-aff7-2c01130e95ee-kube-api-access-8xc4q\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.001003 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/96367807-1ef3-4c30-b605-d041d6f53ae8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:39.001003 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.001102 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001029 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9qh\" (UniqueName: \"kubernetes.io/projected/f17f5606-10aa-4f02-a2c4-16c28aba8540-kube-api-access-9z9qh\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.001102 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.001102 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.000998 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-service-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.001102 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8vg\" (UniqueName: \"kubernetes.io/projected/f67e5454-86f4-4e80-880a-070e4e7a38c1-kube-api-access-wr8vg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.001257 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.001257 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjtb\" (UniqueName: \"kubernetes.io/projected/4334dda9-cc0b-49bd-a5ca-48457df595fe-kube-api-access-6wjtb\") pod \"kas-connection-checker-685895c7c9-ksnkf\" (UID: \"4334dda9-cc0b-49bd-a5ca-48457df595fe\") " pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:20:39.001361 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.001413 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001383 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.001413 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001407 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.001516 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.001516 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001443 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-config\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.001516 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001466 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-trusted-ca\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.001516 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001484 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89b5x\" (UniqueName: \"kubernetes.io/projected/12259580-c199-4cac-a5c2-35a79e28ca90-kube-api-access-89b5x\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.001516 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-snapshots\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.001752 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8jw\" (UniqueName: \"kubernetes.io/projected/f5824bdb-ee34-4c21-8655-5016b9338e5c-kube-api-access-9s8jw\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.001752 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001571 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.001752 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001632 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.001752 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001649 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcp6k\" (UniqueName: \"kubernetes.io/projected/96818867-d92e-402c-b179-d93d211f45a1-kube-api-access-kcp6k\") pod \"network-check-source-6f78d59f99-76bkz\" (UID: \"96818867-d92e-402c-b179-d93d211f45a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:20:39.001752 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12259580-c199-4cac-a5c2-35a79e28ca90-serving-cert\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tww9t\" (UniqueName: \"kubernetes.io/projected/25825c7b-a2c6-4263-bc62-d2c85c00ec47-kube-api-access-tww9t\") pod \"kas-connection-checker-685895c7c9-v4vbz\" (UID: \"25825c7b-a2c6-4263-bc62-d2c85c00ec47\") " pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f67e5454-86f4-4e80-880a-070e4e7a38c1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001817 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d33e27-7f70-4b3c-b90d-927c598a0630-serving-cert\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-default-certificate\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgj2q\" (UniqueName: \"kubernetes.io/projected/52aa29ef-34be-481f-b3a6-d78bfeb0b181-kube-api-access-dgj2q\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f67e5454-86f4-4e80-880a-070e4e7a38c1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.001978 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.001978 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002006 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-snapshots\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.002085 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002126 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.002241 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls podName:52aa29ef-34be-481f-b3a6-d78bfeb0b181 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.502222298 +0000 UTC m=+33.660023439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wnxfz" (UID: "52aa29ef-34be-481f-b3a6-d78bfeb0b181") : secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52aa29ef-34be-481f-b3a6-d78bfeb0b181-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002452 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-stats-auth\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.002463 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.002736 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:39.002736 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.002531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.002736 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.002648 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.502637289 +0000 UTC m=+33.660438431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : configmap references non-existent config key: service-ca.crt Oct 31 18:20:39.002736 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.002714 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:39.002736 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.002722 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8558f787fd-29dwp: secret "image-registry-tls" not found Oct 31 18:20:39.002869 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.002741 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls podName:888afe16-546c-4f4b-a742-2eef09e161ab nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.502735228 +0000 UTC m=+33.660536355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls") pod "image-registry-8558f787fd-29dwp" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab") : secret "image-registry-tls" not found Oct 31 18:20:39.009087 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.008384 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.009087 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.008747 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.009087 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.008758 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-default-certificate\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.009385 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.009114 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52aa29ef-34be-481f-b3a6-d78bfeb0b181-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:39.010521 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.010495 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.010733 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.010718 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d33e27-7f70-4b3c-b90d-927c598a0630-serving-cert\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.010834 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.010819 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-stats-auth\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.058756 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.058725 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrkf\" (UniqueName: \"kubernetes.io/projected/875a3d19-4e6c-4bb6-87b9-e801cd4f6604-kube-api-access-ggrkf\") pod \"kas-connection-checker-685895c7c9-xgpld\" (UID: \"875a3d19-4e6c-4bb6-87b9-e801cd4f6604\") " pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:20:39.058884 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.058833 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrpt\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.059053 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.059029 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclxl\" (UniqueName: \"kubernetes.io/projected/b6d33e27-7f70-4b3c-b90d-927c598a0630-kube-api-access-sclxl\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.059338 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.059316 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.059580 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.059561 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjtb\" (UniqueName: \"kubernetes.io/projected/4334dda9-cc0b-49bd-a5ca-48457df595fe-kube-api-access-6wjtb\") pod \"kas-connection-checker-685895c7c9-ksnkf\" (UID: \"4334dda9-cc0b-49bd-a5ca-48457df595fe\") " pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:20:39.059712 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.059688 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgj2q\" (UniqueName: \"kubernetes.io/projected/52aa29ef-34be-481f-b3a6-d78bfeb0b181-kube-api-access-dgj2q\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:39.059771 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.059747 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9qh\" (UniqueName: \"kubernetes.io/projected/f17f5606-10aa-4f02-a2c4-16c28aba8540-kube-api-access-9z9qh\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.059812 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.059789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tww9t\" (UniqueName: \"kubernetes.io/projected/25825c7b-a2c6-4263-bc62-d2c85c00ec47-kube-api-access-tww9t\") pod \"kas-connection-checker-685895c7c9-v4vbz\" (UID: \"25825c7b-a2c6-4263-bc62-d2c85c00ec47\") " pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:20:39.103399 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103373 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:20:39.103545 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.103616 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:39.103616 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29596\" (UniqueName: \"kubernetes.io/projected/9d5010af-7674-4de3-ac32-4799caaf4f39-kube-api-access-29596\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.103710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.103710 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.103667 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 18:20:39.103710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f957aba3-d3ed-48a0-aff7-2c01130e95ee-config\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.103847 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103753 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.103847 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.103765 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 18:20:39.103847 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.103775 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert podName:96367807-1ef3-4c30-b605-d041d6f53ae8 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.603758511 +0000 UTC m=+33.761559652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xncwq" (UID: "96367807-1ef3-4c30-b605-d041d6f53ae8") : secret "networking-console-plugin-cert" not found Oct 31 18:20:39.103847 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f957aba3-d3ed-48a0-aff7-2c01130e95ee-serving-cert\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.103847 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.103812 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls podName:f5824bdb-ee34-4c21-8655-5016b9338e5c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.60380128 +0000 UTC m=+33.761602422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z7w8t" (UID: "f5824bdb-ee34-4c21-8655-5016b9338e5c") : secret "samples-operator-tls" not found Oct 31 18:20:39.103847 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.104154 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8dt\" (UniqueName: \"kubernetes.io/projected/f58038f2-47ce-43fc-bfa5-116eed7f4776-kube-api-access-lj8dt\") pod \"volume-data-source-validator-58c9f5f46d-98g5c\" (UID: \"f58038f2-47ce-43fc-bfa5-116eed7f4776\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:20:39.104154 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103884 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xc4q\" (UniqueName: \"kubernetes.io/projected/f957aba3-d3ed-48a0-aff7-2c01130e95ee-kube-api-access-8xc4q\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.104154 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.103993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/96367807-1ef3-4c30-b605-d041d6f53ae8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:39.104154 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.104154 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104059 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.104154 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.104147 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.104162 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cbfd9d9f-q2vwz: secret "image-registry-tls" not found Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8vg\" (UniqueName: \"kubernetes.io/projected/f67e5454-86f4-4e80-880a-070e4e7a38c1-kube-api-access-wr8vg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.104226 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls podName:08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.60421493 +0000 UTC m=+33.762016056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls") pod "image-registry-85cbfd9d9f-q2vwz" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5") : secret "image-registry-tls" not found Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6kt\" (UniqueName: \"kubernetes.io/projected/db87f267-b523-4a65-ab4c-b343a2d9ace5-kube-api-access-hd6kt\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104281 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f957aba3-d3ed-48a0-aff7-2c01130e95ee-config\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.104439 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104301 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87f267-b523-4a65-ab4c-b343a2d9ace5-tmp-dir\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.104790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.104790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-config\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.104790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-trusted-ca\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.104790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89b5x\" (UniqueName: \"kubernetes.io/projected/12259580-c199-4cac-a5c2-35a79e28ca90-kube-api-access-89b5x\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.104790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/96367807-1ef3-4c30-b605-d041d6f53ae8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8jw\" (UniqueName: \"kubernetes.io/projected/f5824bdb-ee34-4c21-8655-5016b9338e5c-kube-api-access-9s8jw\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcp6k\" (UniqueName: \"kubernetes.io/projected/96818867-d92e-402c-b179-d93d211f45a1-kube-api-access-kcp6k\") pod \"network-check-source-6f78d59f99-76bkz\" (UID: \"96818867-d92e-402c-b179-d93d211f45a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12259580-c199-4cac-a5c2-35a79e28ca90-serving-cert\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f67e5454-86f4-4e80-880a-070e4e7a38c1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5010af-7674-4de3-ac32-4799caaf4f39-cert\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.104989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f67e5454-86f4-4e80-880a-070e4e7a38c1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db87f267-b523-4a65-ab4c-b343a2d9ace5-config-volume\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105069 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105074 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.105210 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105153 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-config\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.105758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105371 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.105758 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105508 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-trusted-ca\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.106144 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.105875 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f67e5454-86f4-4e80-880a-070e4e7a38c1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.106144 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.106109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.106415 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.106394 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f957aba3-d3ed-48a0-aff7-2c01130e95ee-serving-cert\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.106526 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.106506 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.106764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.106741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.106827 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.106803 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.107526 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.107510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12259580-c199-4cac-a5c2-35a79e28ca90-serving-cert\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.107854 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.107839 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f67e5454-86f4-4e80-880a-070e4e7a38c1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.161129 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.161061 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8vg\" (UniqueName: \"kubernetes.io/projected/f67e5454-86f4-4e80-880a-070e4e7a38c1-kube-api-access-wr8vg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.161300 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.161153 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.161421 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.161400 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.161632 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.161613 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8dt\" (UniqueName: \"kubernetes.io/projected/f58038f2-47ce-43fc-bfa5-116eed7f4776-kube-api-access-lj8dt\") pod \"volume-data-source-validator-58c9f5f46d-98g5c\" (UID: \"f58038f2-47ce-43fc-bfa5-116eed7f4776\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:20:39.161791 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.161774 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xc4q\" (UniqueName: \"kubernetes.io/projected/f957aba3-d3ed-48a0-aff7-2c01130e95ee-kube-api-access-8xc4q\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.163099 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.163077 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8jw\" (UniqueName: \"kubernetes.io/projected/f5824bdb-ee34-4c21-8655-5016b9338e5c-kube-api-access-9s8jw\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.163194 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.163148 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89b5x\" (UniqueName: \"kubernetes.io/projected/12259580-c199-4cac-a5c2-35a79e28ca90-kube-api-access-89b5x\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.163493 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.163477 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcp6k\" (UniqueName: \"kubernetes.io/projected/96818867-d92e-402c-b179-d93d211f45a1-kube-api-access-kcp6k\") pod \"network-check-source-6f78d59f99-76bkz\" (UID: \"96818867-d92e-402c-b179-d93d211f45a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:20:39.172217 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.172176 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:39.205626 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.205604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87f267-b523-4a65-ab4c-b343a2d9ace5-tmp-dir\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.205763 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.205680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5010af-7674-4de3-ac32-4799caaf4f39-cert\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.205763 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.205704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db87f267-b523-4a65-ab4c-b343a2d9ace5-config-volume\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.205763 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.205754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29596\" (UniqueName: \"kubernetes.io/projected/9d5010af-7674-4de3-ac32-4799caaf4f39-kube-api-access-29596\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.205856 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.205835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.205886 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.205863 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6kt\" (UniqueName: \"kubernetes.io/projected/db87f267-b523-4a65-ab4c-b343a2d9ace5-kube-api-access-hd6kt\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.206239 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.206098 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 18:20:39.206239 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.206170 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls podName:db87f267-b523-4a65-ab4c-b343a2d9ace5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:39.706148755 +0000 UTC m=+33.863949887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls") pod "dns-default-pcvwd" (UID: "db87f267-b523-4a65-ab4c-b343a2d9ace5") : secret "dns-default-metrics-tls" not found Oct 31 18:20:39.206239 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.206236 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87f267-b523-4a65-ab4c-b343a2d9ace5-tmp-dir\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.206417 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.206399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db87f267-b523-4a65-ab4c-b343a2d9ace5-config-volume\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.208723 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.208701 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5010af-7674-4de3-ac32-4799caaf4f39-cert\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.219059 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.219040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:20:39.262201 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.262159 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29596\" (UniqueName: \"kubernetes.io/projected/9d5010af-7674-4de3-ac32-4799caaf4f39-kube-api-access-29596\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.262806 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.262787 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6kt\" (UniqueName: \"kubernetes.io/projected/db87f267-b523-4a65-ab4c-b343a2d9ace5-kube-api-access-hd6kt\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.329593 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.329570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:20:39.343671 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.343649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:20:39.357557 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.357529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:20:39.432761 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.432733 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:20:39.447678 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.447657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:20:39.460586 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.460564 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:20:39.503767 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.503738 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8z6s5"] Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.521552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.521704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.521756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.521803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.521978 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.521959589 +0000 UTC m=+34.679760731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : configmap references non-existent config key: service-ca.crt Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522412 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522459 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.522444581 +0000 UTC m=+34.680245721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : secret "router-metrics-certs-default" not found Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522520 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522584 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls podName:52aa29ef-34be-481f-b3a6-d78bfeb0b181 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.52254176 +0000 UTC m=+34.680342901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wnxfz" (UID: "52aa29ef-34be-481f-b3a6-d78bfeb0b181") : secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522644 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522655 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8558f787fd-29dwp: secret "image-registry-tls" not found Oct 31 18:20:39.540176 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.522682 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls podName:888afe16-546c-4f4b-a742-2eef09e161ab nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.522672695 +0000 UTC m=+34.680473822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls") pod "image-registry-8558f787fd-29dwp" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab") : secret "image-registry-tls" not found Oct 31 18:20:39.556532 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.556507 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zqcnd"] Oct 31 18:20:39.559215 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.558718 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12259580_c199_4cac_a5c2_35a79e28ca90.slice/crio-9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9 WatchSource:0}: Error finding container 9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9: Status 404 returned error can't find the container with id 9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9 Oct 31 18:20:39.559851 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.559822 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:20:39.604605 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.604579 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz"] Oct 31 18:20:39.607895 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.607855 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96818867_d92e_402c_b179_d93d211f45a1.slice/crio-9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8 WatchSource:0}: Error finding container 9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8: Status 404 returned error can't find the container with id 9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8 Oct 31 18:20:39.623113 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.623088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:39.623224 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.623140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:39.623291 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.623230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:39.623413 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623398 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:39.623468 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623415 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cbfd9d9f-q2vwz: secret "image-registry-tls" not found Oct 31 18:20:39.623468 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623454 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls podName:08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.62344194 +0000 UTC m=+34.781243070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls") pod "image-registry-85cbfd9d9f-q2vwz" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5") : secret "image-registry-tls" not found Oct 31 18:20:39.623593 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623501 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 18:20:39.623593 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623522 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert podName:96367807-1ef3-4c30-b605-d041d6f53ae8 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.623516405 +0000 UTC m=+34.781317531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xncwq" (UID: "96367807-1ef3-4c30-b605-d041d6f53ae8") : secret "networking-console-plugin-cert" not found Oct 31 18:20:39.623593 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623567 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 18:20:39.623593 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.623585 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls podName:f5824bdb-ee34-4c21-8655-5016b9338e5c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.623579853 +0000 UTC m=+34.781380980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z7w8t" (UID: "f5824bdb-ee34-4c21-8655-5016b9338e5c") : secret "samples-operator-tls" not found Oct 31 18:20:39.711733 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.711623 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgpld"] Oct 31 18:20:39.715076 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.715050 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875a3d19_4e6c_4bb6_87b9_e801cd4f6604.slice/crio-bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1 WatchSource:0}: Error finding container bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1: Status 404 returned error can't find the container with id bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1 Oct 31 18:20:39.724597 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.724577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:39.724769 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.724750 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 18:20:39.724835 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:39.724820 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls podName:db87f267-b523-4a65-ab4c-b343a2d9ace5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:40.724805272 +0000 UTC m=+34.882606411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls") pod "dns-default-pcvwd" (UID: "db87f267-b523-4a65-ab4c-b343a2d9ace5") : secret "dns-default-metrics-tls" not found Oct 31 18:20:39.737533 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.737501 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v4vbz"] Oct 31 18:20:39.740604 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.740580 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ksnkf"] Oct 31 18:20:39.741730 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.741650 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25825c7b_a2c6_4263_bc62_d2c85c00ec47.slice/crio-3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4 WatchSource:0}: Error finding container 3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4: Status 404 returned error can't find the container with id 3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4 Oct 31 18:20:39.743946 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.743916 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4334dda9_cc0b_49bd_a5ca_48457df595fe.slice/crio-a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7 WatchSource:0}: Error finding container a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7: Status 404 returned error can't find the container with id a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7 Oct 31 18:20:39.823872 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.823847 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c"] Oct 31 18:20:39.832609 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.832583 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58038f2_47ce_43fc_bfa5_116eed7f4776.slice/crio-9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8 WatchSource:0}: Error finding container 9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8: Status 404 returned error can't find the container with id 9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8 Oct 31 18:20:39.872975 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.872946 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v"] Oct 31 18:20:39.876466 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.876420 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf957aba3_d3ed_48a0_aff7_2c01130e95ee.slice/crio-aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7 WatchSource:0}: Error finding container aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7: Status 404 returned error can't find the container with id aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7 Oct 31 18:20:39.877322 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.877303 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm"] Oct 31 18:20:39.884756 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.884735 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67e5454_86f4_4e80_880a_070e4e7a38c1.slice/crio-e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a WatchSource:0}: Error finding container e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a: Status 404 returned error can't find the container with id e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a Oct 31 18:20:39.939997 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:39.939974 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-px79l"] Oct 31 18:20:39.943280 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:39.943256 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5010af_7674_4de3_ac32_4799caaf4f39.slice/crio-1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78 WatchSource:0}: Error finding container 1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78: Status 404 returned error can't find the container with id 1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78 Oct 31 18:20:40.229155 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.229122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:40.229386 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.229166 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:40.229386 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.229283 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:40.229386 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.229335 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:40.229386 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.229339 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs podName:f22f6aaa-22cc-4ab9-a4bb-90d73eac861c nodeName:}" failed. No retries permitted until 2026-10-31 18:21:12.229326003 +0000 UTC m=+66.387127130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs") pod "network-metrics-daemon-xkth6" (UID: "f22f6aaa-22cc-4ab9-a4bb-90d73eac861c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 18:20:40.229508 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.229391 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret podName:96e03cbb-2909-4802-accd-002dd2922c42 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:12.229378756 +0000 UTC m=+66.387179883 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret") pod "global-pull-secret-syncer-tczvl" (UID: "96e03cbb-2909-4802-accd-002dd2922c42") : object "kube-system"/"original-pull-secret" not registered Oct 31 18:20:40.300623 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.300598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:40.300745 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.300711 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:20:40.300818 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.300803 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:20:40.330355 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.330334 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:40.332909 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.332890 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:40.353084 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.353060 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 18:20:40.353084 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.353061 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 18:20:40.353228 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.353132 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\"" Oct 31 18:20:40.353228 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.353157 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\"" Oct 31 18:20:40.353228 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.353178 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\"" Oct 31 18:20:40.360386 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.360370 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:40.477532 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.477494 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-px79l" event={"ID":"9d5010af-7674-4de3-ac32-4799caaf4f39","Type":"ContainerStarted","Data":"1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78"} Oct 31 18:20:40.478954 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.478929 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" event={"ID":"f67e5454-86f4-4e80-880a-070e4e7a38c1","Type":"ContainerStarted","Data":"e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a"} Oct 31 18:20:40.480589 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.480526 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xgpld" event={"ID":"875a3d19-4e6c-4bb6-87b9-e801cd4f6604","Type":"ContainerStarted","Data":"9558a1a76135d49c9196e6a9abdddcadc9c0dd2e71d63182c2ad6196e033aba8"} Oct 31 18:20:40.480589 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.480572 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xgpld" event={"ID":"875a3d19-4e6c-4bb6-87b9-e801cd4f6604","Type":"ContainerStarted","Data":"bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1"} Oct 31 18:20:40.481957 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.481939 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" event={"ID":"96818867-d92e-402c-b179-d93d211f45a1","Type":"ContainerStarted","Data":"9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8"} Oct 31 18:20:40.483093 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.483071 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" event={"ID":"f957aba3-d3ed-48a0-aff7-2c01130e95ee","Type":"ContainerStarted","Data":"aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7"} Oct 31 18:20:40.484458 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.484430 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" event={"ID":"b6d33e27-7f70-4b3c-b90d-927c598a0630","Type":"ContainerStarted","Data":"6a9778c230d77be0c6a477517a4fa4cd3b913455c7a8dfffeac8a3f50f92e04c"} Oct 31 18:20:40.485664 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.485645 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" event={"ID":"f58038f2-47ce-43fc-bfa5-116eed7f4776","Type":"ContainerStarted","Data":"9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8"} Oct 31 18:20:40.486981 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.486959 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" event={"ID":"12259580-c199-4cac-a5c2-35a79e28ca90","Type":"ContainerStarted","Data":"9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9"} Oct 31 18:20:40.488333 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.488310 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" event={"ID":"4334dda9-cc0b-49bd-a5ca-48457df595fe","Type":"ContainerStarted","Data":"4005c6350a022674ed00a29a30e9bf731b70b975ffaec995977f3776ec47a36c"} Oct 31 18:20:40.488441 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.488353 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" event={"ID":"4334dda9-cc0b-49bd-a5ca-48457df595fe","Type":"ContainerStarted","Data":"a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7"} Oct 31 18:20:40.490018 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.489996 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" event={"ID":"25825c7b-a2c6-4263-bc62-d2c85c00ec47","Type":"ContainerStarted","Data":"944baf8d4eb87fada0bc90803393f6450ab097dbb1615dc434d39b6181271e34"} Oct 31 18:20:40.490106 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.490034 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" event={"ID":"25825c7b-a2c6-4263-bc62-d2c85c00ec47","Type":"ContainerStarted","Data":"3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4"} Oct 31 18:20:40.532525 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.532496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:40.532642 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.532556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:40.532642 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.532610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:40.532728 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532685 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 18:20:40.532759 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532754 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.532734012 +0000 UTC m=+36.690535151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : secret "router-metrics-certs-default" not found Oct 31 18:20:40.532796 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532774 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:40.532796 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532782 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.532761414 +0000 UTC m=+36.690562553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : configmap references non-existent config key: service-ca.crt Oct 31 18:20:40.532907 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532789 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8558f787fd-29dwp: secret "image-registry-tls" not found Oct 31 18:20:40.532907 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.532893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:40.532977 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532954 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls podName:888afe16-546c-4f4b-a742-2eef09e161ab nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.532887799 +0000 UTC m=+36.690688926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls") pod "image-registry-8558f787fd-29dwp" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab") : secret "image-registry-tls" not found Oct 31 18:20:40.533024 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.532995 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:40.533060 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.533036 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls podName:52aa29ef-34be-481f-b3a6-d78bfeb0b181 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.533023619 +0000 UTC m=+36.690824760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wnxfz" (UID: "52aa29ef-34be-481f-b3a6-d78bfeb0b181") : secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:40.633967 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.633938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:40.634120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.633992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:40.634120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.634082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:40.634120 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634098 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 18:20:40.634332 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634162 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert podName:96367807-1ef3-4c30-b605-d041d6f53ae8 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.634144024 +0000 UTC m=+36.791945168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xncwq" (UID: "96367807-1ef3-4c30-b605-d041d6f53ae8") : secret "networking-console-plugin-cert" not found Oct 31 18:20:40.634332 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634279 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 18:20:40.634332 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634315 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls podName:f5824bdb-ee34-4c21-8655-5016b9338e5c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.634304838 +0000 UTC m=+36.792105988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z7w8t" (UID: "f5824bdb-ee34-4c21-8655-5016b9338e5c") : secret "samples-operator-tls" not found Oct 31 18:20:40.634887 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634867 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:40.634887 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634888 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cbfd9d9f-q2vwz: secret "image-registry-tls" not found Oct 31 18:20:40.635021 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.634923 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls podName:08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.634911819 +0000 UTC m=+36.792712945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls") pod "image-registry-85cbfd9d9f-q2vwz" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5") : secret "image-registry-tls" not found Oct 31 18:20:40.735422 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.735389 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:40.735643 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.735621 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 18:20:40.735760 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:40.735677 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls podName:db87f267-b523-4a65-ab4c-b343a2d9ace5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:42.735664749 +0000 UTC m=+36.893465888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls") pod "dns-default-pcvwd" (UID: "db87f267-b523-4a65-ab4c-b343a2d9ace5") : secret "dns-default-metrics-tls" not found Oct 31 18:20:40.747583 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.747556 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bch5w"] Oct 31 18:20:40.751768 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:40.751736 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfe1fd5_4835_4b2b_87d6_d218cfcd6b85.slice/crio-848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c WatchSource:0}: Error finding container 848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c: Status 404 returned error can't find the container with id 848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c Oct 31 18:20:40.785268 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:40.784565 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" podStartSLOduration=437.784547286 podStartE2EDuration="7m17.784547286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:40.7830793 +0000 UTC m=+34.940880449" watchObservedRunningTime="2026-10-31 18:20:40.784547286 +0000 UTC m=+34.942348435" Oct 31 18:20:41.009054 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:41.008781 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xgpld" podStartSLOduration=438.00876289 podStartE2EDuration="7m18.00876289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:40.896762165 +0000 UTC m=+35.054563315" watchObservedRunningTime="2026-10-31 18:20:41.00876289 +0000 UTC m=+35.166564057" Oct 31 18:20:41.500278 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:41.500240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bch5w" event={"ID":"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85","Type":"ContainerStarted","Data":"848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c"} Oct 31 18:20:42.555399 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.555364 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:42.555822 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.555495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:42.555822 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.555544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:42.555822 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.555592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:42.555822 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555761 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:42.555822 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555787 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 18:20:42.555822 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555827 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls podName:52aa29ef-34be-481f-b3a6-d78bfeb0b181 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.555808213 +0000 UTC m=+40.713609355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wnxfz" (UID: "52aa29ef-34be-481f-b3a6-d78bfeb0b181") : secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:42.556116 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555853 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.55583425 +0000 UTC m=+40.713635413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : secret "router-metrics-certs-default" not found Oct 31 18:20:42.556116 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555916 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:42.556116 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555930 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8558f787fd-29dwp: secret "image-registry-tls" not found Oct 31 18:20:42.556116 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.555961 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls podName:888afe16-546c-4f4b-a742-2eef09e161ab nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.555950617 +0000 UTC m=+40.713751775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls") pod "image-registry-8558f787fd-29dwp" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab") : secret "image-registry-tls" not found Oct 31 18:20:42.556325 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.556149 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.556133441 +0000 UTC m=+40.713934568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : configmap references non-existent config key: service-ca.crt Oct 31 18:20:42.657668 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.657623 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:42.657824 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.657715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:42.657824 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.657795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:42.658056 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658039 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:42.658099 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658061 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cbfd9d9f-q2vwz: secret "image-registry-tls" not found Oct 31 18:20:42.658131 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658115 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls podName:08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.65809715 +0000 UTC m=+40.815898297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls") pod "image-registry-85cbfd9d9f-q2vwz" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5") : secret "image-registry-tls" not found Oct 31 18:20:42.658390 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658218 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 18:20:42.658390 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658263 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert podName:96367807-1ef3-4c30-b605-d041d6f53ae8 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.658251142 +0000 UTC m=+40.816052272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xncwq" (UID: "96367807-1ef3-4c30-b605-d041d6f53ae8") : secret "networking-console-plugin-cert" not found Oct 31 18:20:42.658390 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658322 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 18:20:42.658390 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.658353 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls podName:f5824bdb-ee34-4c21-8655-5016b9338e5c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.658343347 +0000 UTC m=+40.816144486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z7w8t" (UID: "f5824bdb-ee34-4c21-8655-5016b9338e5c") : secret "samples-operator-tls" not found Oct 31 18:20:42.758643 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:42.758608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:42.758843 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.758822 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 18:20:42.758915 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:42.758887 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls podName:db87f267-b523-4a65-ab4c-b343a2d9ace5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.758873424 +0000 UTC m=+40.916674553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls") pod "dns-default-pcvwd" (UID: "db87f267-b523-4a65-ab4c-b343a2d9ace5") : secret "dns-default-metrics-tls" not found Oct 31 18:20:43.512229 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:43.511817 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" event={"ID":"f58038f2-47ce-43fc-bfa5-116eed7f4776","Type":"ContainerStarted","Data":"99bdae7b65cd5bed02c1529696564ef3f63ddcad90ff115314757a123022ca49"} Oct 31 18:20:43.626205 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:43.626133 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" podStartSLOduration=440.626119786 podStartE2EDuration="7m20.626119786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:41.009345259 +0000 UTC m=+35.167146409" watchObservedRunningTime="2026-10-31 18:20:43.626119786 +0000 UTC m=+37.783920933" Oct 31 18:20:43.626657 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:43.626636 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" podStartSLOduration=409.01106264 podStartE2EDuration="6m51.626630013s" totalImagesPullingTime="2.615567373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:43.624926467 +0000 UTC m=+37.782727614" watchObservedRunningTime="2026-10-31 18:20:43.626630013 +0000 UTC m=+37.784431157" Oct 31 18:20:44.519146 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:44.518389 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" event={"ID":"b6d33e27-7f70-4b3c-b90d-927c598a0630","Type":"ContainerStarted","Data":"9d817cb6345d8060b037d37cc11b381dd002bd41574077b0cf7e14b98e94ed47"} Oct 31 18:20:44.521031 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:44.521004 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/0.log" Oct 31 18:20:44.521156 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:44.521042 2572 generic.go:356] "Generic (PLEG): container finished" podID="12259580-c199-4cac-a5c2-35a79e28ca90" containerID="d0c67327dd89bebf6e7b2d9f72daadfc59114fe97606fddbf97b80242115926d" exitCode=255 Oct 31 18:20:44.521754 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:44.521738 2572 scope.go:122] "RemoveContainer" containerID="d0c67327dd89bebf6e7b2d9f72daadfc59114fe97606fddbf97b80242115926d" Oct 31 18:20:44.522011 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:44.521994 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" event={"ID":"12259580-c199-4cac-a5c2-35a79e28ca90","Type":"ContainerDied","Data":"d0c67327dd89bebf6e7b2d9f72daadfc59114fe97606fddbf97b80242115926d"} Oct 31 18:20:44.650328 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:44.650273 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" podStartSLOduration=408.160283297 podStartE2EDuration="6m52.650251476s" totalImagesPullingTime="4.489968179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:44.632933845 +0000 UTC m=+38.790734996" watchObservedRunningTime="2026-10-31 18:20:44.650251476 +0000 UTC m=+38.808052624" Oct 31 18:20:45.425330 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.425289 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-np8hr"] Oct 31 18:20:45.444361 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.444338 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.487633 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.487607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25de07e5-55c8-4c35-96af-7861b0a74469-crio-socket\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.487782 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.487641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltprw\" (UniqueName: \"kubernetes.io/projected/25de07e5-55c8-4c35-96af-7861b0a74469-kube-api-access-ltprw\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.487782 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.487659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25de07e5-55c8-4c35-96af-7861b0a74469-data-volume\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.487782 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.487717 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.487926 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.487806 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.487926 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.487870 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-np8hr"] Oct 31 18:20:45.526794 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.526760 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/1.log" Oct 31 18:20:45.527535 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.527516 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/0.log" Oct 31 18:20:45.527654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.527555 2572 generic.go:356] "Generic (PLEG): container finished" podID="12259580-c199-4cac-a5c2-35a79e28ca90" containerID="bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d" exitCode=255 Oct 31 18:20:45.527654 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.527642 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" event={"ID":"12259580-c199-4cac-a5c2-35a79e28ca90","Type":"ContainerDied","Data":"bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d"} Oct 31 18:20:45.527737 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.527690 2572 scope.go:122] "RemoveContainer" containerID="d0c67327dd89bebf6e7b2d9f72daadfc59114fe97606fddbf97b80242115926d" Oct 31 18:20:45.528095 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.528072 2572 scope.go:122] "RemoveContainer" containerID="bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d" Oct 31 18:20:45.528313 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:45.528292 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zqcnd_openshift-console-operator(12259580-c199-4cac-a5c2-35a79e28ca90)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podUID="12259580-c199-4cac-a5c2-35a79e28ca90" Oct 31 18:20:45.535270 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.535251 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 18:20:45.535367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.535314 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 18:20:45.535367 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.535330 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\"" Oct 31 18:20:45.588362 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.588326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.588516 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:45.588458 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 18:20:45.588516 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.588485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.588621 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:45.588524 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.088505018 +0000 UTC m=+40.246306171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : configmap "kube-rbac-proxy" not found Oct 31 18:20:45.588621 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:45.588590 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 18:20:45.588621 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.588600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25de07e5-55c8-4c35-96af-7861b0a74469-crio-socket\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.588767 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:45.588649 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.088624239 +0000 UTC m=+40.246425381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : secret "insights-runtime-extractor-tls" not found Oct 31 18:20:45.588767 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.588668 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltprw\" (UniqueName: \"kubernetes.io/projected/25de07e5-55c8-4c35-96af-7861b0a74469-kube-api-access-ltprw\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.588767 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.588699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25de07e5-55c8-4c35-96af-7861b0a74469-data-volume\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.588900 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.588792 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25de07e5-55c8-4c35-96af-7861b0a74469-crio-socket\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.589422 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.589399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25de07e5-55c8-4c35-96af-7861b0a74469-data-volume\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:45.650359 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:45.650326 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltprw\" (UniqueName: \"kubernetes.io/projected/25de07e5-55c8-4c35-96af-7861b0a74469-kube-api-access-ltprw\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:46.093781 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.093583 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:46.093781 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.093691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:46.093781 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.093714 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 18:20:46.093781 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.093782 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:47.093765516 +0000 UTC m=+41.251566658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : configmap "kube-rbac-proxy" not found Oct 31 18:20:46.094050 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.093838 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 18:20:46.094050 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.093897 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:47.093883118 +0000 UTC m=+41.251684258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : secret "insights-runtime-extractor-tls" not found Oct 31 18:20:46.532856 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.532822 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-px79l" event={"ID":"9d5010af-7674-4de3-ac32-4799caaf4f39","Type":"ContainerStarted","Data":"1a46127d4c1065006a21e8fbaac210ebb2ae5ec1119154560d90d1eaf401bfe9"} Oct 31 18:20:46.534404 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.534379 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" event={"ID":"96818867-d92e-402c-b179-d93d211f45a1","Type":"ContainerStarted","Data":"0d3d90c6ce3b19638557a4fc156d91ac0fdb23b2888464ef7c4d8ad6b0c84644"} Oct 31 18:20:46.535996 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.535971 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bch5w" event={"ID":"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85","Type":"ContainerStarted","Data":"e55fb70a141a6ce4785c079c8c1ac45964f2bbebd368b879296c3882a13afe40"} Oct 31 18:20:46.536096 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.536081 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:20:46.537530 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.537500 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" event={"ID":"f957aba3-d3ed-48a0-aff7-2c01130e95ee","Type":"ContainerStarted","Data":"35e74d9b3c83ca33dddf9a71efe99540d5d87bc12c840b201d3f378ea97e2157"} Oct 31 18:20:46.538963 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.538949 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/1.log" Oct 31 18:20:46.539264 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.539249 2572 scope.go:122] "RemoveContainer" containerID="bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d" Oct 31 18:20:46.539438 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.539419 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zqcnd_openshift-console-operator(12259580-c199-4cac-a5c2-35a79e28ca90)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podUID="12259580-c199-4cac-a5c2-35a79e28ca90" Oct 31 18:20:46.598928 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.598657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:46.598928 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.598839 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:46.598928 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.598895 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls podName:52aa29ef-34be-481f-b3a6-d78bfeb0b181 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.598878779 +0000 UTC m=+48.756679921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wnxfz" (UID: "52aa29ef-34be-481f-b3a6-d78bfeb0b181") : secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:46.599124 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.598946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:46.599124 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.599006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:46.599124 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.599069 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:46.599124 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.599076 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:46.599124 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.599084 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8558f787fd-29dwp: secret "image-registry-tls" not found Oct 31 18:20:46.599359 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.599138 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls podName:888afe16-546c-4f4b-a742-2eef09e161ab nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.599122653 +0000 UTC m=+48.756923788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls") pod "image-registry-8558f787fd-29dwp" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab") : secret "image-registry-tls" not found Oct 31 18:20:46.599359 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.599229 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.599219108 +0000 UTC m=+48.757020236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : configmap references non-existent config key: service-ca.crt Oct 31 18:20:46.599359 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.599304 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 18:20:46.599359 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.599341 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.599331833 +0000 UTC m=+48.757132983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : secret "router-metrics-certs-default" not found Oct 31 18:20:46.644521 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.644475 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-px79l" podStartSLOduration=2.6496284 podStartE2EDuration="8.644461804s" totalImagesPullingTime="5.994833404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:46.643424001 +0000 UTC m=+40.801225149" watchObservedRunningTime="2026-10-31 18:20:46.644461804 +0000 UTC m=+40.802262950" Oct 31 18:20:46.700109 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.700071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.700163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700245 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.700273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700303 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert podName:96367807-1ef3-4c30-b605-d041d6f53ae8 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.700286723 +0000 UTC m=+48.858087866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xncwq" (UID: "96367807-1ef3-4c30-b605-d041d6f53ae8") : secret "networking-console-plugin-cert" not found Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700324 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700347 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700358 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cbfd9d9f-q2vwz: secret "image-registry-tls" not found Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700387 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls podName:f5824bdb-ee34-4c21-8655-5016b9338e5c nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.700371319 +0000 UTC m=+48.858172470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z7w8t" (UID: "f5824bdb-ee34-4c21-8655-5016b9338e5c") : secret "samples-operator-tls" not found Oct 31 18:20:46.700560 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.700481 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls podName:08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.700471794 +0000 UTC m=+48.858272924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls") pod "image-registry-85cbfd9d9f-q2vwz" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5") : secret "image-registry-tls" not found Oct 31 18:20:46.755448 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.755406 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" podStartSLOduration=406.442065459 podStartE2EDuration="6m52.755391533s" totalImagesPullingTime="6.313326074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:46.754260297 +0000 UTC m=+40.912061446" watchObservedRunningTime="2026-10-31 18:20:46.755391533 +0000 UTC m=+40.913192707" Oct 31 18:20:46.801086 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.801027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:46.801222 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.801167 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 18:20:46.801288 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:46.801275 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls podName:db87f267-b523-4a65-ab4c-b343a2d9ace5 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.801249063 +0000 UTC m=+48.959050189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls") pod "dns-default-pcvwd" (UID: "db87f267-b523-4a65-ab4c-b343a2d9ace5") : secret "dns-default-metrics-tls" not found Oct 31 18:20:46.869046 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.868977 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bch5w" podStartSLOduration=34.436256302 podStartE2EDuration="39.868960362s" totalImagesPullingTime="5.43270406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:46.866841338 +0000 UTC m=+41.024642486" watchObservedRunningTime="2026-10-31 18:20:46.868960362 +0000 UTC m=+41.026761512" Oct 31 18:20:46.984245 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:46.984168 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" podStartSLOduration=408.937479265 podStartE2EDuration="6m54.984150109s" totalImagesPullingTime="6.046670844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:46.983247606 +0000 UTC m=+41.141048755" watchObservedRunningTime="2026-10-31 18:20:46.984150109 +0000 UTC m=+41.141951256" Oct 31 18:20:47.104034 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:47.103957 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:47.104223 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:47.104064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:47.104223 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:47.104094 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 18:20:47.104223 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:47.104173 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 18:20:47.104223 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:47.104196 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:49.104161999 +0000 UTC m=+43.261963139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : configmap "kube-rbac-proxy" not found Oct 31 18:20:47.104376 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:47.104230 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:49.104218585 +0000 UTC m=+43.262019731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : secret "insights-runtime-extractor-tls" not found Oct 31 18:20:49.125533 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.125501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:49.125914 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.125631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:49.125914 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:49.125674 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 18:20:49.125914 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:49.125696 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 18:20:49.125914 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:49.125744 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:53.125727228 +0000 UTC m=+47.283528367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : configmap "kube-rbac-proxy" not found Oct 31 18:20:49.125914 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:49.125760 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:53.125752057 +0000 UTC m=+47.283553186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : secret "insights-runtime-extractor-tls" not found Oct 31 18:20:49.172723 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.172689 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:49.172866 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.172749 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:49.173094 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.173082 2572 scope.go:122] "RemoveContainer" containerID="bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d" Oct 31 18:20:49.173293 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:49.173275 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zqcnd_openshift-console-operator(12259580-c199-4cac-a5c2-35a79e28ca90)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podUID="12259580-c199-4cac-a5c2-35a79e28ca90" Oct 31 18:20:49.549490 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.549455 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" event={"ID":"f67e5454-86f4-4e80-880a-070e4e7a38c1","Type":"ContainerStarted","Data":"584f8d5e7733aca9d5f3041c2524ecd48dd285e990c4fa5dc7f0580a68e29a76"} Oct 31 18:20:49.675974 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:49.675919 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" podStartSLOduration=409.101858509 podStartE2EDuration="6m57.675905088s" totalImagesPullingTime="8.574046579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:49.675075367 +0000 UTC m=+43.832876517" watchObservedRunningTime="2026-10-31 18:20:49.675905088 +0000 UTC m=+43.833706260" Oct 31 18:20:53.160975 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:53.160939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:53.161565 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:53.161043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:53.161565 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:53.161154 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 18:20:53.161565 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:53.161252 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:01.161237644 +0000 UTC m=+55.319038771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : secret "insights-runtime-extractor-tls" not found Oct 31 18:20:53.161731 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:53.161645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:20:53.785641 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:53.785615 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z89h2_8ce43312-6761-45de-8350-d347681cb539/dns-node-resolver/0.log" Oct 31 18:20:54.590998 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.590970 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hf5q2_d0906409-dea7-4779-9d86-4c9bfee0ee91/node-ca/0.log" Oct 31 18:20:54.673792 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.673762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:20:54.673950 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.673804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:20:54.673950 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.673896 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:54.673950 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.673900 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:54.673950 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.673906 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8558f787fd-29dwp: secret "image-registry-tls" not found Oct 31 18:20:54.673950 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.673949 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls podName:52aa29ef-34be-481f-b3a6-d78bfeb0b181 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.673936585 +0000 UTC m=+64.831737711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wnxfz" (UID: "52aa29ef-34be-481f-b3a6-d78bfeb0b181") : secret "cluster-monitoring-operator-tls" not found Oct 31 18:20:54.674228 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.673964 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls podName:888afe16-546c-4f4b-a742-2eef09e161ab nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.67395831 +0000 UTC m=+64.831759436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls") pod "image-registry-8558f787fd-29dwp" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab") : secret "image-registry-tls" not found Oct 31 18:20:54.674228 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.673940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:54.674228 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.674028 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.674012879 +0000 UTC m=+64.831814024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : configmap references non-existent config key: service-ca.crt Oct 31 18:20:54.674228 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.674097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:20:54.674228 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.674226 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 18:20:54.674407 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.674271 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs podName:f17f5606-10aa-4f02-a2c4-16c28aba8540 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.6742607 +0000 UTC m=+64.832061826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs") pod "router-default-59bdcd5559-qvgdn" (UID: "f17f5606-10aa-4f02-a2c4-16c28aba8540") : secret "router-metrics-certs-default" not found Oct 31 18:20:54.774872 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.774843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:20:54.775007 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.774895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:20:54.775007 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.774940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:20:54.775007 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.774977 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 18:20:54.775130 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.775025 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 18:20:54.775130 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.775034 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cbfd9d9f-q2vwz: secret "image-registry-tls" not found Oct 31 18:20:54.775130 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.775037 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert podName:96367807-1ef3-4c30-b605-d041d6f53ae8 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.775023009 +0000 UTC m=+64.932824149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xncwq" (UID: "96367807-1ef3-4c30-b605-d041d6f53ae8") : secret "networking-console-plugin-cert" not found Oct 31 18:20:54.775130 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.775046 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 18:20:54.775130 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.775072 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls podName:08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.775062039 +0000 UTC m=+64.932863165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls") pod "image-registry-85cbfd9d9f-q2vwz" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5") : secret "image-registry-tls" not found Oct 31 18:20:54.775130 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:54.775086 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls podName:f5824bdb-ee34-4c21-8655-5016b9338e5c nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.775076621 +0000 UTC m=+64.932877751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z7w8t" (UID: "f5824bdb-ee34-4c21-8655-5016b9338e5c") : secret "samples-operator-tls" not found Oct 31 18:20:54.876368 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.876299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:54.878632 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:54.878610 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:55.121421 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.121390 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-px79l_9d5010af-7674-4de3-ac32-4799caaf4f39/serve-healthcheck-canary/0.log" Oct 31 18:20:55.174003 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.173937 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcvwd" Oct 31 18:20:55.297398 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.297371 2572 scope.go:122] "RemoveContainer" containerID="bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d" Oct 31 18:20:55.523122 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.523093 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm_f67e5454-86f4-4e80-880a-070e4e7a38c1/kube-storage-version-migrator-operator/0.log" Oct 31 18:20:55.540024 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.540002 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcvwd"] Oct 31 18:20:55.542951 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:20:55.542927 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb87f267_b523_4a65_ab4c_b343a2d9ace5.slice/crio-bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc WatchSource:0}: Error finding container bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc: Status 404 returned error can't find the container with id bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc Oct 31 18:20:55.566649 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.566624 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcvwd" event={"ID":"db87f267-b523-4a65-ab4c-b343a2d9ace5","Type":"ContainerStarted","Data":"bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc"} Oct 31 18:20:55.568010 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.567991 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:20:55.568447 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.568431 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/1.log" Oct 31 18:20:55.568529 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.568465 2572 generic.go:356] "Generic (PLEG): container finished" podID="12259580-c199-4cac-a5c2-35a79e28ca90" containerID="ca740c27ab199d9f4f1113e32e9acd6fa6571ec20bd2a9aabec12fb15a5fe80c" exitCode=255 Oct 31 18:20:55.568529 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.568520 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" event={"ID":"12259580-c199-4cac-a5c2-35a79e28ca90","Type":"ContainerDied","Data":"ca740c27ab199d9f4f1113e32e9acd6fa6571ec20bd2a9aabec12fb15a5fe80c"} Oct 31 18:20:55.568631 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.568570 2572 scope.go:122] "RemoveContainer" containerID="bd254becbe63b4a1bc5b8ac4593e3e037c8d77da14e5650d046a4f9aee345b4d" Oct 31 18:20:55.568955 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:55.568937 2572 scope.go:122] "RemoveContainer" containerID="ca740c27ab199d9f4f1113e32e9acd6fa6571ec20bd2a9aabec12fb15a5fe80c" Oct 31 18:20:55.569129 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:55.569111 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zqcnd_openshift-console-operator(12259580-c199-4cac-a5c2-35a79e28ca90)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podUID="12259580-c199-4cac-a5c2-35a79e28ca90" Oct 31 18:20:56.574145 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:56.574108 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:20:59.172303 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:59.172266 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:59.172787 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:59.172336 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:20:59.172787 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:20:59.172693 2572 scope.go:122] "RemoveContainer" containerID="ca740c27ab199d9f4f1113e32e9acd6fa6571ec20bd2a9aabec12fb15a5fe80c" Oct 31 18:20:59.172930 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:20:59.172905 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zqcnd_openshift-console-operator(12259580-c199-4cac-a5c2-35a79e28ca90)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podUID="12259580-c199-4cac-a5c2-35a79e28ca90" Oct 31 18:21:01.227853 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:01.227808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:01.228328 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:21:01.227960 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 18:21:01.228328 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:21:01.228025 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls podName:25de07e5-55c8-4c35-96af-7861b0a74469 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:17.228010989 +0000 UTC m=+71.385812116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np8hr" (UID: "25de07e5-55c8-4c35-96af-7861b0a74469") : secret "insights-runtime-extractor-tls" not found Oct 31 18:21:02.598585 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:02.598547 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcvwd" event={"ID":"db87f267-b523-4a65-ab4c-b343a2d9ace5","Type":"ContainerStarted","Data":"bd878135f23548f947770887ea2ebe1a3f1f173aeffa5ee2d5d40da9fb647e82"} Oct 31 18:21:02.599057 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:02.598601 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcvwd" event={"ID":"db87f267-b523-4a65-ab4c-b343a2d9ace5","Type":"ContainerStarted","Data":"9a97a04d41ca1cd212006389569d2fba2081b4d37041b894a10ceeb872687e8c"} Oct 31 18:21:02.599057 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:02.598678 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:02.713701 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:02.713659 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pcvwd" podStartSLOduration=18.103993288 podStartE2EDuration="24.713644563s" totalImagesPullingTime="6.609651275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:02.712026327 +0000 UTC m=+56.869827472" watchObservedRunningTime="2026-10-31 18:21:02.713644563 +0000 UTC m=+56.871445711" Oct 31 18:21:03.473613 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:03.473585 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:10.270137 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.270096 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2"] Oct 31 18:21:10.272907 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.272890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:10.318709 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.318684 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8558f787fd-29dwp"] Oct 31 18:21:10.318848 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:21:10.318831 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-8558f787fd-29dwp" podUID="888afe16-546c-4f4b-a742-2eef09e161ab" Oct 31 18:21:10.327930 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.327909 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:10.340242 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.340221 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 18:21:10.345594 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.345569 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2"] Oct 31 18:21:10.406066 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.406043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7t7\" (UniqueName: \"kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7\") pod \"migrator-66df5d496b-gpzw2\" (UID: \"c9621db0-ac02-43c2-9edd-ec7bd05d1817\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:10.417403 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.417375 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gl4z9\"" Oct 31 18:21:10.507153 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.507125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7t7\" (UniqueName: \"kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7\") pod \"migrator-66df5d496b-gpzw2\" (UID: \"c9621db0-ac02-43c2-9edd-ec7bd05d1817\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:10.566805 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:21:10.566743 2572 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found Oct 31 18:21:10.566805 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:21:10.566764 2572 projected.go:196] Error preparing data for projected volume kube-api-access-qj7t7 for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2: configmap "openshift-service-ca.crt" not found Oct 31 18:21:10.566942 ip-10-0-140-47 kubenswrapper[2572]: E1031 18:21:10.566822 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7 podName:c9621db0-ac02-43c2-9edd-ec7bd05d1817 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:11.066807701 +0000 UTC m=+65.224608839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qj7t7" (UniqueName: "kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7") pod "migrator-66df5d496b-gpzw2" (UID: "c9621db0-ac02-43c2-9edd-ec7bd05d1817") : configmap "openshift-service-ca.crt" not found Oct 31 18:21:10.622140 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.622117 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:21:10.627491 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.627459 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:21:10.709115 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709087 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709282 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709145 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709282 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709175 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709282 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709237 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709282 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709264 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709477 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709309 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709477 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709336 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.709477 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:10.709614 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:10.709661 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:21:10.709710 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:10.709759 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709740 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:21:10.709805 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.709749 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:21:10.710852 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.710171 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:21:10.710852 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.710600 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:10.711857 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.711818 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:21:10.711857 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.711836 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:21:10.712144 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.712119 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "kube-api-access-fxrpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:21:10.712144 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.712126 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:21:10.712673 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.712627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:10.712856 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.712834 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:10.713079 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.713063 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"image-registry-8558f787fd-29dwp\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:21:10.810781 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.810750 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") pod \"888afe16-546c-4f4b-a742-2eef09e161ab\" (UID: \"888afe16-546c-4f4b-a742-2eef09e161ab\") " Oct 31 18:21:10.810959 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.810941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:10.811074 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:10.811128 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:10.811178 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811153 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxrpt\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-kube-api-access-fxrpt\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.811178 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811163 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-registry-certificates\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.811178 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811173 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-installation-pull-secrets\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.811178 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811250 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/888afe16-546c-4f4b-a742-2eef09e161ab-trusted-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.811477 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811261 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-bound-sa-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.811477 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811270 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/888afe16-546c-4f4b-a742-2eef09e161ab-ca-trust-extracted\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.811477 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.811278 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/888afe16-546c-4f4b-a742-2eef09e161ab-image-registry-private-configuration\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.813000 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.812976 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls" pod "888afe16-546c-4f4b-a742-2eef09e161ab" (UID: "888afe16-546c-4f4b-a742-2eef09e161ab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:21:10.813414 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.813391 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:10.813499 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.813395 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:10.813544 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.813512 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:10.911973 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.911940 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/888afe16-546c-4f4b-a742-2eef09e161ab-registry-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:10.937243 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.937204 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6pq8w\"" Oct 31 18:21:10.946339 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.946319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:10.970031 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.970006 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-52br2\"" Oct 31 18:21:10.979733 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:10.979710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:11.049209 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.049172 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7pnqt\"" Oct 31 18:21:11.056276 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.056257 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p4b7h\"" Oct 31 18:21:11.059400 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.059380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:11.066120 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.066104 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:11.085809 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.085786 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sqcvs\"" Oct 31 18:21:11.096062 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.096035 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:11.113741 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.113712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7t7\" (UniqueName: \"kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7\") pod \"migrator-66df5d496b-gpzw2\" (UID: \"c9621db0-ac02-43c2-9edd-ec7bd05d1817\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:11.116777 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.116725 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7t7\" (UniqueName: \"kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7\") pod \"migrator-66df5d496b-gpzw2\" (UID: \"c9621db0-ac02-43c2-9edd-ec7bd05d1817\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:11.182444 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.182412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:11.366473 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.366402 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz"] Oct 31 18:21:11.369072 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.369047 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59bdcd5559-qvgdn"] Oct 31 18:21:11.369240 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:11.369216 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aa29ef_34be_481f_b3a6_d78bfeb0b181.slice/crio-2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122 WatchSource:0}: Error finding container 2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122: Status 404 returned error can't find the container with id 2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122 Oct 31 18:21:11.419286 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:11.419262 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17f5606_10aa_4f02_a2c4_16c28aba8540.slice/crio-7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041 WatchSource:0}: Error finding container 7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041: Status 404 returned error can't find the container with id 7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041 Oct 31 18:21:11.478790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.478758 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xncwq"] Oct 31 18:21:11.482245 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:11.482220 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96367807_1ef3_4c30_b605_d041d6f53ae8.slice/crio-b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d WatchSource:0}: Error finding container b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d: Status 404 returned error can't find the container with id b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d Oct 31 18:21:11.482555 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.482538 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t"] Oct 31 18:21:11.483806 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.483777 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85cbfd9d9f-q2vwz"] Oct 31 18:21:11.568031 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.568000 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2"] Oct 31 18:21:11.570865 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:11.570839 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9621db0_ac02_43c2_9edd_ec7bd05d1817.slice/crio-a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef WatchSource:0}: Error finding container a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef: Status 404 returned error can't find the container with id a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef Oct 31 18:21:11.627546 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.627467 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" event={"ID":"f5824bdb-ee34-4c21-8655-5016b9338e5c","Type":"ContainerStarted","Data":"c3cc3b19bdc8ae69614f74ad1db5446a9aa604115f69c07c00e847d6f02c2ba9"} Oct 31 18:21:11.629173 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.629146 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" event={"ID":"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5","Type":"ContainerStarted","Data":"6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7"} Oct 31 18:21:11.629332 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.629197 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" event={"ID":"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5","Type":"ContainerStarted","Data":"2da5603647dd25425daee5a88755a7a281087a098101bbb80ffcf1307d6dc717"} Oct 31 18:21:11.629332 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.629312 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:11.630450 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.630424 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" event={"ID":"96367807-1ef3-4c30-b605-d041d6f53ae8","Type":"ContainerStarted","Data":"b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d"} Oct 31 18:21:11.631913 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.631886 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" event={"ID":"f17f5606-10aa-4f02-a2c4-16c28aba8540","Type":"ContainerStarted","Data":"585290d74ba25860dd46efb21497d963ac1203c3c296fe8d9d635de45f293b7e"} Oct 31 18:21:11.632039 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.631932 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" event={"ID":"f17f5606-10aa-4f02-a2c4-16c28aba8540","Type":"ContainerStarted","Data":"7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041"} Oct 31 18:21:11.632974 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.632956 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" event={"ID":"c9621db0-ac02-43c2-9edd-ec7bd05d1817","Type":"ContainerStarted","Data":"a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef"} Oct 31 18:21:11.634175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.634153 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8558f787fd-29dwp" Oct 31 18:21:11.634175 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.634164 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" event={"ID":"52aa29ef-34be-481f-b3a6-d78bfeb0b181","Type":"ContainerStarted","Data":"2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122"} Oct 31 18:21:11.745712 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.745655 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" podStartSLOduration=56.745640999 podStartE2EDuration="56.745640999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:11.744155976 +0000 UTC m=+65.901957124" watchObservedRunningTime="2026-10-31 18:21:11.745640999 +0000 UTC m=+65.903442146" Oct 31 18:21:11.915946 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.915876 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8558f787fd-29dwp"] Oct 31 18:21:11.919959 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.919938 2572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8558f787fd-29dwp"] Oct 31 18:21:11.980634 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.980609 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:11.982824 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:11.982805 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:12.035341 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.035294 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" podStartSLOduration=440.035276634 podStartE2EDuration="7m20.035276634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:12.033779153 +0000 UTC m=+66.191580300" watchObservedRunningTime="2026-10-31 18:21:12.035276634 +0000 UTC m=+66.193077784" Oct 31 18:21:12.302411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.302382 2572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888afe16-546c-4f4b-a742-2eef09e161ab" path="/var/lib/kubelet/pods/888afe16-546c-4f4b-a742-2eef09e161ab/volumes" Oct 31 18:21:12.324655 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.324627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:12.324790 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.324661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:12.375685 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.375653 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 18:21:12.375685 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.375653 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 18:21:12.387927 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.387899 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:12.388012 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.387998 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:12.560560 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.560490 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\"" Oct 31 18:21:12.560560 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.560554 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:12.574535 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.574484 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\"" Oct 31 18:21:12.574649 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.574621 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:12.606691 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.606512 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:12.648923 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.648820 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:12.650417 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.650345 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:12.950855 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.950825 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkth6"] Oct 31 18:21:12.953283 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:12.953241 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22f6aaa_22cc_4ab9_a4bb_90d73eac861c.slice/crio-4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59 WatchSource:0}: Error finding container 4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59: Status 404 returned error can't find the container with id 4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59 Oct 31 18:21:12.983003 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:12.982968 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tczvl"] Oct 31 18:21:12.983983 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:12.983956 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e03cbb_2909_4802_accd_002dd2922c42.slice/crio-97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610 WatchSource:0}: Error finding container 97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610: Status 404 returned error can't find the container with id 97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610 Oct 31 18:21:13.653411 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:13.653134 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" event={"ID":"96367807-1ef3-4c30-b605-d041d6f53ae8","Type":"ContainerStarted","Data":"167531cae5d6a80dd01272c5660176b72bbf76450cb80bb46beb134e343a4e41"} Oct 31 18:21:13.654764 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:13.654737 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" event={"ID":"c9621db0-ac02-43c2-9edd-ec7bd05d1817","Type":"ContainerStarted","Data":"c43003dcdd68d130dda588ebc7f3836046ced50cbb326bb7c01b2ae45a656d0b"} Oct 31 18:21:13.655941 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:13.655915 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tczvl" event={"ID":"96e03cbb-2909-4802-accd-002dd2922c42","Type":"ContainerStarted","Data":"97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610"} Oct 31 18:21:13.657247 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:13.657222 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkth6" event={"ID":"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c","Type":"ContainerStarted","Data":"4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59"} Oct 31 18:21:13.766319 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:13.766256 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" podStartSLOduration=404.829895665 podStartE2EDuration="6m46.766237475s" totalImagesPullingTime="1.93634181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:13.764291994 +0000 UTC m=+67.922093143" watchObservedRunningTime="2026-10-31 18:21:13.766237475 +0000 UTC m=+67.924038624" Oct 31 18:21:13.876915 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:13.876854 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" podStartSLOduration=1.921726802 podStartE2EDuration="3.876840109s" totalImagesPullingTime="1.955113307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:13.875524387 +0000 UTC m=+68.033325535" watchObservedRunningTime="2026-10-31 18:21:13.876840109 +0000 UTC m=+68.034641257" Oct 31 18:21:14.663656 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:14.663492 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" event={"ID":"52aa29ef-34be-481f-b3a6-d78bfeb0b181","Type":"ContainerStarted","Data":"96a7db900b3b4b651aa6aecad296a4c5748d8eabf107c0ff14a621d033e4cc5a"} Oct 31 18:21:14.668107 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:14.667449 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" event={"ID":"f5824bdb-ee34-4c21-8655-5016b9338e5c","Type":"ContainerStarted","Data":"023706233eb2c49e6a92d5779d178f7d3ff4d5bb2bc63439c3795a6acf258592"} Oct 31 18:21:14.668107 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:14.667502 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" event={"ID":"f5824bdb-ee34-4c21-8655-5016b9338e5c","Type":"ContainerStarted","Data":"2031470861e0f0aff7d3eaf4fc3f3ee2c5a2b8d0e3ed6a1ac2e35869b5e1b850"} Oct 31 18:21:14.782104 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:14.782050 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" podStartSLOduration=439.918541609 podStartE2EDuration="7m22.78203273s" totalImagesPullingTime="2.863491121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:14.781904712 +0000 UTC m=+68.939705861" watchObservedRunningTime="2026-10-31 18:21:14.78203273 +0000 UTC m=+68.939833879" Oct 31 18:21:14.899051 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:14.898993 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" podStartSLOduration=440.011494676 podStartE2EDuration="7m22.898979839s" totalImagesPullingTime="2.887485163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:14.897096376 +0000 UTC m=+69.054897523" watchObservedRunningTime="2026-10-31 18:21:14.898979839 +0000 UTC m=+69.056780981" Oct 31 18:21:15.671430 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:15.671391 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkth6" event={"ID":"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c","Type":"ContainerStarted","Data":"ca03a3d63866e4ec65c57e444f7bd7174ce3814714e4217dabc8e9a54580b4bc"} Oct 31 18:21:15.671838 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:15.671451 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkth6" event={"ID":"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c","Type":"ContainerStarted","Data":"a2ac791658f6459949aabe2757af56b6a3827784814ab56a926af39d0ef153d2"} Oct 31 18:21:15.786378 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:15.786322 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xkth6" podStartSLOduration=67.138043757 podStartE2EDuration="1m8.78630785s" totalImagesPullingTime="1.648264093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:15.785267849 +0000 UTC m=+69.943068991" watchObservedRunningTime="2026-10-31 18:21:15.78630785 +0000 UTC m=+69.944109004" Oct 31 18:21:17.265899 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:17.265862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:17.268639 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:17.268613 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:17.372872 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:17.372841 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\"" Oct 31 18:21:17.382949 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:17.382922 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:17.544317 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:17.544251 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:21:17.789106 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:17.789076 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-np8hr"] Oct 31 18:21:17.792556 ip-10-0-140-47 kubenswrapper[2572]: W1031 18:21:17.792526 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25de07e5_55c8_4c35_96af_7861b0a74469.slice/crio-bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7 WatchSource:0}: Error finding container bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7: Status 404 returned error can't find the container with id bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7 Oct 31 18:21:18.297515 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.297480 2572 scope.go:122] "RemoveContainer" containerID="ca740c27ab199d9f4f1113e32e9acd6fa6571ec20bd2a9aabec12fb15a5fe80c" Oct 31 18:21:18.685946 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.685887 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:21:18.686093 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.686004 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" event={"ID":"12259580-c199-4cac-a5c2-35a79e28ca90","Type":"ContainerStarted","Data":"612d9dd751122bbc0b9b0c56d0b2f36561f3f2b801e3da2293de05da159329cf"} Oct 31 18:21:18.686336 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.686315 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:18.687521 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.687494 2572 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zqcnd container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body= Oct 31 18:21:18.687613 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.687551 2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podUID="12259580-c199-4cac-a5c2-35a79e28ca90" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" Oct 31 18:21:18.687664 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.687623 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np8hr" event={"ID":"25de07e5-55c8-4c35-96af-7861b0a74469","Type":"ContainerStarted","Data":"471dc35013d8abda8b86b72d878f42cb79f7c015495d66267248cf1bdb7d1d6a"} Oct 31 18:21:18.687706 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.687660 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np8hr" event={"ID":"25de07e5-55c8-4c35-96af-7861b0a74469","Type":"ContainerStarted","Data":"bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7"} Oct 31 18:21:18.804917 ip-10-0-140-47 kubenswrapper[2572]: I1031 18:21:18.804872 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" podStartSLOduration=442.34428794 podStartE2EDuration="7m26.804856852s" totalImagesPullingTime="4.460568912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:18.803717694 +0000 UTC m=+72.961518853" watchObservedRunningTime="2026-10-31 18:21:18.804856852 +0000 UTC m=+72.962658000" Oct 31 18:21:19.338653 ip-10-0-140-47 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 18:21:19.354574 ip-10-0-140-47 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 18:21:19.354737 ip-10-0-140-47 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 18:21:19.354786 ip-10-0-140-47 systemd[1]: kubelet.service: Consumed 4.206s CPU time. Oct 31 18:21:19.365647 ip-10-0-140-47 systemd[1]: Starting Kubernetes Kubelet... Oct 31 18:21:19.450889 ip-10-0-140-47 kubenswrapper[5688]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 18:21:19.450889 ip-10-0-140-47 kubenswrapper[5688]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 18:21:19.450889 ip-10-0-140-47 kubenswrapper[5688]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 18:21:19.450889 ip-10-0-140-47 kubenswrapper[5688]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 18:21:19.453059 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.452976 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.453059 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453056 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.453059 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453064 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453068 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453072 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453077 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453081 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453085 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453089 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453092 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453096 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453102 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453109 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453113 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453118 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453122 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453128 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453131 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453135 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453139 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453159 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.453306 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453164 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453169 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453173 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453177 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453199 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453203 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453207 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453212 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453216 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453220 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453224 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453228 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453232 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453237 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453241 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453245 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453249 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453253 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453257 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453261 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.454153 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453265 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453269 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453273 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453277 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453281 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453285 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453289 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453293 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453297 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453301 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453305 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453309 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453321 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453325 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453331 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453335 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453340 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453344 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453349 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453353 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.455001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453357 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453361 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453365 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453370 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453374 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453378 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453382 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453386 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453390 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453394 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453398 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453402 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453406 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453410 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453414 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453418 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453423 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453427 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453431 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453435 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.455893 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453439 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453445 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453451 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453458 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453463 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453475 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453480 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453484 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453490 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453494 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453498 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453502 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453506 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453511 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453515 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453519 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453524 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453528 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453532 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.456802 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453536 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453540 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453543 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453547 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453551 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.453555 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454820 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454830 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454834 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454839 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454844 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454848 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454852 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454858 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454862 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454866 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454870 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454874 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454878 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.457674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454883 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454888 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454892 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454896 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454900 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454904 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454908 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454912 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454916 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454921 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454925 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454930 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454934 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454939 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454943 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454947 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454951 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454955 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454959 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454963 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.458605 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454967 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454972 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454976 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454980 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454984 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454988 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454992 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.454997 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455000 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455004 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455008 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455011 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455015 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455027 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455032 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455036 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455040 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455044 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455048 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.459452 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455052 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455057 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455061 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455065 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455069 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455073 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455078 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455090 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455094 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455098 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455102 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455106 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455111 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455115 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455119 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455123 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455130 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455137 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455142 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.460001 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455147 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455152 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455157 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455161 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455165 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455170 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455174 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455199 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455211 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455216 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455221 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455225 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455229 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455232 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455236 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455241 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455248 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455252 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455256 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455260 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.460494 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455264 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455268 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455272 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455276 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455280 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455284 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455288 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455293 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455297 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455301 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455330 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455338 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455343 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455348 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.455352 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456204 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456215 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456220 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456224 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.460978 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456227 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456239 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456243 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456248 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456252 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456256 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456261 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456265 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456270 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456277 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456281 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456285 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456289 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456294 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456298 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456302 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456306 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456310 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456314 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.461516 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456318 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456323 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456328 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456332 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456336 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456340 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456344 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456348 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456352 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456356 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456363 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456368 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456373 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456377 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456381 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456392 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456397 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456401 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456405 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.462249 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456409 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456413 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456418 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456423 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456427 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456431 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456435 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456440 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456444 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456448 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456452 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456456 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456460 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456464 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456468 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456472 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456476 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456480 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456484 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456488 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.462984 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456492 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456496 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456500 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456504 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456508 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456512 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456516 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456520 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456523 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456535 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456540 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456544 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456548 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456552 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456556 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456562 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456566 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456570 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456574 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.463493 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456578 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456582 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456586 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456590 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456594 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456598 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456603 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456607 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456618 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456622 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456626 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456630 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456634 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456639 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456643 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456647 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456651 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456655 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456658 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.463950 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456662 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456666 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456670 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456674 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456689 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.456694 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456825 5688 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456837 5688 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456851 5688 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456860 5688 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456867 5688 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456872 5688 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456879 5688 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456885 5688 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456890 5688 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456895 5688 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456900 5688 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456905 5688 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456910 5688 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456915 5688 flags.go:64] FLAG: --cgroup-root="" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456919 5688 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456924 5688 flags.go:64] FLAG: --client-ca-file="" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456928 5688 flags.go:64] FLAG: --cloud-provider="external" Oct 31 18:21:19.464422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456933 5688 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456942 5688 flags.go:64] FLAG: --cluster-domain="" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456947 5688 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456951 5688 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456956 5688 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456962 5688 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456968 5688 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456973 5688 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456977 5688 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456983 5688 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456987 5688 flags.go:64] FLAG: --contention-profiling="false" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456992 5688 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.456996 5688 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457002 5688 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457014 5688 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457021 5688 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457026 5688 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457031 5688 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457037 5688 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457041 5688 flags.go:64] FLAG: --enable-server="true" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457046 5688 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457060 5688 flags.go:64] FLAG: --event-burst="100" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457065 5688 flags.go:64] FLAG: --event-qps="50" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457070 5688 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 18:21:19.464982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457074 5688 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457079 5688 flags.go:64] FLAG: --eviction-hard="" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457085 5688 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457089 5688 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457094 5688 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457099 5688 flags.go:64] FLAG: --eviction-soft="" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457104 5688 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457108 5688 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457113 5688 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457117 5688 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457122 5688 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457126 5688 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457131 5688 flags.go:64] FLAG: --feature-gates="" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457137 5688 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457142 5688 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457147 5688 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457152 5688 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457156 5688 flags.go:64] FLAG: --healthz-port="10248" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457161 5688 flags.go:64] FLAG: --help="false" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457166 5688 flags.go:64] FLAG: --hostname-override="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457173 5688 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457197 5688 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457203 5688 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 18:21:19.465576 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457216 5688 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457222 5688 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457226 5688 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457232 5688 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457237 5688 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457241 5688 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457245 5688 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457251 5688 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457255 5688 flags.go:64] FLAG: --kube-reserved="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457260 5688 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457264 5688 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457269 5688 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457274 5688 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457278 5688 flags.go:64] FLAG: --lock-file="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457282 5688 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457287 5688 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457292 5688 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457300 5688 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457305 5688 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457310 5688 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457314 5688 flags.go:64] FLAG: --logging-format="text" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457319 5688 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457324 5688 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457328 5688 flags.go:64] FLAG: --manifest-url="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457332 5688 flags.go:64] FLAG: --manifest-url-header="" Oct 31 18:21:19.466109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457339 5688 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457345 5688 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457351 5688 flags.go:64] FLAG: --max-pods="110" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457355 5688 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457361 5688 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457366 5688 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457370 5688 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457375 5688 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457387 5688 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457392 5688 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457405 5688 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457410 5688 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457414 5688 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457419 5688 flags.go:64] FLAG: --pod-cidr="" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457423 5688 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457427 5688 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457432 5688 flags.go:64] FLAG: --pods-per-core="0" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457437 5688 flags.go:64] FLAG: --port="10250" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457441 5688 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457446 5688 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-076b9146017615332" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457451 5688 flags.go:64] FLAG: --qos-reserved="" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457456 5688 flags.go:64] FLAG: --read-only-port="10255" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457460 5688 flags.go:64] FLAG: --register-node="true" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457465 5688 flags.go:64] FLAG: --register-with-taints="" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457471 5688 flags.go:64] FLAG: --registry-burst="10" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457476 5688 flags.go:64] FLAG: --registry-qps="5" Oct 31 18:21:19.466845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457480 5688 flags.go:64] FLAG: --reserved-cpus="" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457484 5688 flags.go:64] FLAG: --reserved-memory="" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457490 5688 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457495 5688 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457499 5688 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457504 5688 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457508 5688 flags.go:64] FLAG: --runonce="false" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457513 5688 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457518 5688 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457522 5688 flags.go:64] FLAG: --seccomp-default="false" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457528 5688 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457533 5688 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457542 5688 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457546 5688 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457551 5688 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457563 5688 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457569 5688 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457574 5688 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457578 5688 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457583 5688 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457587 5688 flags.go:64] FLAG: --system-cgroups="" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457592 5688 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457609 5688 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457614 5688 flags.go:64] FLAG: --tls-cert-file="" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457619 5688 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 18:21:19.467581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457628 5688 flags.go:64] FLAG: --tls-min-version="" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457633 5688 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457637 5688 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457641 5688 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457646 5688 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457650 5688 flags.go:64] FLAG: --v="2" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457657 5688 flags.go:64] FLAG: --version="false" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457663 5688 flags.go:64] FLAG: --vmodule="" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457669 5688 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.457673 5688 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457943 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457949 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457954 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457958 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457963 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457967 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457971 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457977 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457981 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457986 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457990 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.457994 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.468277 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458000 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458015 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458020 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458024 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458028 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458032 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458036 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458040 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458044 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458048 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458052 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458056 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458060 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458065 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458069 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458073 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458077 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458081 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458085 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458088 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.468995 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458093 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458097 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458101 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458105 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458109 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458113 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458117 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458123 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458127 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458131 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458135 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458140 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458144 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458148 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458159 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458164 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458170 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458176 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458200 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458205 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.469641 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458209 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458213 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458217 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458221 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458225 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458229 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458233 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458237 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458241 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458245 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458248 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458252 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458256 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458260 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458264 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458268 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458272 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458276 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458280 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.470266 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458285 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458290 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458296 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458301 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458306 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458311 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458315 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458320 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458334 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458338 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458342 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458346 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458350 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458355 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458359 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458363 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458367 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458371 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458375 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.470961 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458379 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458383 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458387 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458392 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458395 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458400 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458404 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458409 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458413 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458417 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458422 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458426 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458430 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458435 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458439 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458443 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.471696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.458487 5688 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458559 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458566 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458572 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458576 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458581 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458592 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458596 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458600 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458604 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458608 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458612 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458615 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458620 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458623 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458627 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458631 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458637 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458642 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458646 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.472448 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458650 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458654 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458659 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458662 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458673 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458678 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458682 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458686 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458690 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458696 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458700 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458704 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458708 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458712 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458716 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458720 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458725 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458728 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458733 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458744 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.473226 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458748 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458752 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458755 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458759 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458763 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458767 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458771 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458776 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458780 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458785 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458789 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458793 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458797 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458802 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458805 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458810 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458814 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458818 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458822 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458826 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.473960 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458830 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458839 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458845 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458850 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458854 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458858 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458862 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458867 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458871 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458875 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458879 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458883 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458894 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458899 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458904 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458908 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458912 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458916 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.474741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458920 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458923 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458927 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458934 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458938 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458942 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458946 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458950 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458956 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458962 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458966 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458970 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458974 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458979 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458982 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458988 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458992 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.458996 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459001 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.475317 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459004 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459009 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459013 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459017 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459021 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459025 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459029 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459033 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459038 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.459044 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.464224 5688 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.464239 5688 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.464258 5688 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.464263 5688 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464334 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464339 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464342 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464345 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464348 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464351 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.475854 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464353 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464356 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464359 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464362 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464364 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464367 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464369 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464371 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464374 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464377 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464380 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464383 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464386 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464389 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464391 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464393 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464396 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464399 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464401 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464404 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.476491 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464407 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464410 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464412 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464415 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464417 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464420 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464423 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464425 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464428 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464430 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464433 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464435 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464438 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464440 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464443 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464445 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464448 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464450 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464454 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.476962 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464458 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464461 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464464 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464467 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464469 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464472 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464474 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464476 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464479 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464481 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464484 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464486 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464489 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464491 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464494 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464498 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464501 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464504 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464507 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.477465 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464510 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464512 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464515 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464517 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464520 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464523 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464525 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464527 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464529 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464532 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464534 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464537 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464539 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464541 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464543 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464546 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464549 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464552 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464554 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464556 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.477907 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464559 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464561 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464564 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464566 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464569 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464571 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464573 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464576 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464579 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464581 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464583 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464586 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464588 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464590 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464593 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464595 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464598 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464600 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464602 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.478633 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464605 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464607 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464609 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.464636 5688 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464782 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464788 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464791 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464793 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464796 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464800 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464803 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464805 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464808 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464811 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464814 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464816 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.479648 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464818 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464821 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464823 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464825 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464828 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464831 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464834 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464836 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464840 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464845 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464848 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464850 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464853 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464855 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464857 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464860 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464862 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464865 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464867 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.480409 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464869 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464871 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464874 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464876 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464879 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464881 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464883 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464886 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464888 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464891 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464893 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464895 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464898 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464900 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464902 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464905 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464907 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464910 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464912 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.481246 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464914 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464917 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464919 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464922 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464924 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464926 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464929 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464931 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464934 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464936 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464938 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464941 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464943 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464946 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464948 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464951 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464953 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464955 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464958 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.482013 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464960 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464962 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464966 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464969 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464973 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464975 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464978 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464980 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464983 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464986 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464989 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464992 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464994 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464997 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.464999 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465002 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465004 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465006 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465009 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.482878 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465011 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465013 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465016 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465018 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465021 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465024 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465026 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465028 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465031 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465033 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465035 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465038 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465040 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465042 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465045 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465047 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465050 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.483691 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465052 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.465075 5688 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465108 5688 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465112 5688 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465115 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465118 5688 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465120 5688 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465123 5688 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465125 5688 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465128 5688 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465130 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465133 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465135 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465137 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465140 5688 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465142 5688 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 18:21:19.484445 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465144 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465151 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465153 5688 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465156 5688 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465158 5688 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465161 5688 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465163 5688 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465165 5688 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465168 5688 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465172 5688 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465176 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465193 5688 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465197 5688 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465201 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465205 5688 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465209 5688 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465214 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465217 5688 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465219 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 18:21:19.485199 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465222 5688 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465224 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465227 5688 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465229 5688 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465233 5688 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465235 5688 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465238 5688 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465240 5688 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465243 5688 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465245 5688 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465248 5688 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465250 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465253 5688 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465255 5688 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465259 5688 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465261 5688 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465264 5688 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465267 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465269 5688 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465272 5688 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 18:21:19.486072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465274 5688 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465276 5688 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465279 5688 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465281 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465284 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465286 5688 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465289 5688 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465291 5688 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465294 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465296 5688 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465299 5688 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465301 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465304 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465306 5688 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465309 5688 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465311 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465317 5688 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465320 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465322 5688 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 18:21:19.487165 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465325 5688 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465328 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465330 5688 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465333 5688 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465335 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465339 5688 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465343 5688 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465347 5688 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465350 5688 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465352 5688 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465355 5688 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465357 5688 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465360 5688 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465362 5688 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465365 5688 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465367 5688 feature_gate.go:386] unrecognized feature gate: Example Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465370 5688 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465372 5688 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465375 5688 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 18:21:19.488048 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465377 5688 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465380 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465382 5688 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465384 5688 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465387 5688 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465390 5688 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465392 5688 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465394 5688 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465398 5688 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465401 5688 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465404 5688 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465407 5688 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465409 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465411 5688 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:19.465414 5688 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.465461 5688 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.466382 5688 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.466440 5688 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.466648 5688 server.go:1014] "Starting client certificate rotation" Oct 31 18:21:19.489014 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.466794 5688 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.466857 5688 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 18:15:06 +0000 UTC" deadline="2028-04-21 11:10:39.226050065 +0000 UTC" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.466892 5688 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12904h49m19.759159836s" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.467218 5688 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.468291 5688 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.469642 5688 log.go:25] "Validated CRI v1 runtime API" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.474358 5688 log.go:25] "Validated CRI v1 image API" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.475646 5688 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 18:21:19.489901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.477809 5688 fs.go:135] Filesystem UUIDs: map[4bc14f40-7879-4892-9345-58abe23559d1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 edc1c76c-d85d-4082-8bb1-61227a4498b0:/dev/nvme0n1p4] Oct 31 18:21:19.490346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.477830 5688 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2da5603647dd25425daee5a88755a7a281087a098101bbb80ffcf1307d6dc717/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2da5603647dd25425daee5a88755a7a281087a098101bbb80ffcf1307d6dc717/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59/userdata/shm major:0 minor:394 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a9778c230d77be0c6a477517a4fa4cd3b913455c7a8dfffeac8a3f50f92e04c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a9778c230d77be0c6a477517a4fa4cd3b913455c7a8dfffeac8a3f50f92e04c/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610/userdata/shm major:0 minor:396 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3cc3b19bdc8ae69614f74ad1db5446a9aa604115f69c07c00e847d6f02c2ba9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3cc3b19bdc8ae69614f74ad1db5446a9aa604115f69c07c00e847d6f02c2ba9/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volumes/kubernetes.io~projected/kube-api-access-bp6xm:{mountpoint:/var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volumes/kubernetes.io~projected/kube-api-access-bp6xm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/kube-api-access-k2vvm:{mountpoint:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/kube-api-access-k2vvm major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/registry-tls major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12259580-c199-4cac-a5c2-35a79e28ca90/volumes/kubernetes.io~projected/kube-api-access-89b5x:{mountpoint:/var/lib/kubelet/pods/12259580-c199-4cac-a5c2-35a79e28ca90/volumes/kubernetes.io~projected/kube-api-access-89b5x major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12259580-c199-4cac-a5c2-35a79e28ca90/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/12259580-c199-4cac-a5c2-35a79e28ca90/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25825c7b-a2c6-4263-bc62-d2c85c00ec47/volumes/kubernetes.io~projected/kube-api-access-tww9t:{mountpoint:/var/lib/kubelet/pods/25825c7b-a2c6-4263-bc62-d2c85c00ec47/volumes/kubernetes.io~projected/kube-api-access-tww9t major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25de07e5-55c8-4c35-96af-7861b0a74469/volumes/kubernetes.io~projected/kube-api-access-ltprw:{mountpoint:/var/lib/kubelet/pods/25de07e5-55c8-4c35-96af-7861b0a74469/volumes/kubernetes.io~projected/kube-api-access-ltprw major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25de07e5-55c8-4c35-96af-7861b0a74469/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/25de07e5-55c8-4c35-96af-7861b0a74469/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~projected/kube-api-access-wbr4h:{mountpoint:/var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~projected/kube-api-access-wbr4h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4334dda9-cc0b-49bd-a5ca-48457df595fe/volumes/kubernetes.io~projected/kube-api-access-6wjtb:{mountpoint:/var/lib/kubelet/pods/4334dda9-cc0b-49bd-a5ca-48457df595fe/volumes/kubernetes.io~projected/kube-api-access-6wjtb major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/474a81eb-340b-45fe-8033-0deaecf35f03/volumes/kubernetes.io~projected/kube-api-access-csg5w:{mountpoint:/var/lib/kubelet/pods/474a81eb-340b-45fe-8033-0deaecf35f03/volumes/kubernetes.io~projected/kube-api-access-csg5w major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/512c0bfc-c41f-4abd-b4cd-c85d358fac3e/volumes/kubernetes.io~projected/kube-api-access-n4qp6:{mountpoint:/var/lib/kubelet/pods/512c0bfc-c41f-4abd-b4cd-c85d358fac3e/volumes/kubernetes.io~projected/kube-api-access-n4qp6 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52aa29ef-34be-481f-b3a6-d78bfeb0b181/volumes/kubernetes.io~projected/kube-api-access-dgj2q:{mountpoint:/var/lib/kubelet/pods/52aa29ef-34be-481f-b3a6-d78bfeb0b181/volumes/kubernetes.io~projected/kube-api-access-dgj2q major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52aa29ef-34be-481f-b3a6-d78bfeb0b181/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/52aa29ef-34be-481f-b3a6-d78bfeb0b181/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759/volumes/kubernetes.io~projected/kube-api-access-xh4nj:{mountpoint:/var/lib/kubelet/pods/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759/volumes/kubernetes.io~projected/kube-api-access-xh4nj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/875a3d19-4e6c-4bb6-87b9-e801cd4f6604/volumes/kubernetes.io~projected/kube-api-access-ggrkf:{mountpoint:/var/lib/kubelet/pods/875a3d19-4e6c-4bb6-87b9-e801cd4f6604/volumes/kubernetes.io~projected/kube-api-access-ggrkf major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89f9210d-bf91-47c5-a4ab-318f8da1985d/volumes/kubernetes.io~projected/kube-api-access-nnd79:{mountpoint:/var/lib/kubelet/pods/89f9210d-bf91-47c5-a4ab-318f8da1985d/volumes/kubernetes.io~projected/kube-api-access-nnd79 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ce43312-6761-45de-8350-d347681cb539/volumes/kubernetes.io~projected/kube-api-access-5zn6x:{mountpoint:/var/lib/kubelet/pods/8ce43312-6761-45de-8350-d347681cb539/volumes/kubernetes.io~projected/kube-api-access-5zn6x major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96367807-1ef3-4c30-b605-d041d6f53ae8/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/96367807-1ef3-4c30-b605-d041d6f53ae8/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96818867-d92e-402c-b179-d93d211f45a1/volumes/kubernetes.io~projected/kube-api-access-kcp6k:{mountpoint:/var/lib/kubelet/pods/96818867-d92e-402c-b179-d93d211f45a1/volumes/kubernetes.io~projected/kube-api-access-kcp6k major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e03cbb-2909-4802-accd-002dd2922c42/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/96e03cbb-2909-4802-accd-002dd2922c42/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d5010af-7674-4de3-ac32-4799caaf4f39/volumes/kubernetes.io~projected/kube-api-access-29596:{mountpoint:/var/lib/kubelet/pods/9d5010af-7674-4de3-ac32-4799caaf4f39/volumes/kubernetes.io~projected/kube-api-access-29596 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d5010af-7674-4de3-ac32-4799caaf4f39/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9d5010af-7674-4de3-ac32-4799caaf4f39/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85/volumes/kubernetes.io~projected/kube-api-access-4xpmx:{mountpoint:/var/lib/kubelet/pods/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85/volumes/kubernetes.io~projected/kube-api-access-4xpmx major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d33e27-7f70-4b3c-b90d-927c598a0630/volumes/kubernetes.io~projected/kube-api-access-sclxl:{mountpoint:/var/lib/kubelet/pods/b6d33e27-7f70-4b3c-b90d-927c598a0630/volumes/kubernetes.io~projected/kube-api-access-sclxl major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d33e27-7f70-4b3c-b90d-927c598a0630/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b6d33e27-7f70-4b3c-b90d-927c598a0630/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9621db0-ac02-43c2-9edd-ec7bd05d1817/volumes/kubernetes.io~projected/kube-api-access-qj7t7:{mountpoint:/var/lib/kubelet/pods/c9621db0-ac02-43c2-9edd-ec7bd05d1817/volumes/kubernetes.io~projected/kube-api-access-qj7t7 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc74a8fe-8053-4b78-b8bf-569e5e34a89f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cc74a8fe-8053-4b78-b8bf-569e5e34a89f/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0906409-dea7-4779-9d86-4c9bfee0ee91/volumes/kubernetes.io~projected/kube-api-access-cksmw:{mountpoint:/var/lib/kubelet/pods/d0906409-dea7-4779-9d86-4c9bfee0ee91/volumes/kubernetes.io~projected/kube-api-access-cksmw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db87f267-b523-4a65-ab4c-b343a2d9ace5/volumes/kubernetes.io~projected/kube-api-access-hd6kt:{mountpoint:/var/lib/kubelet/pods/db87f267-b523-4a65-ab4c-b343a2d9ace5/volumes/kubernetes.io~projected/kube-api-access-hd6kt major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db87f267-b523-4a65-ab4c-b343a2d9ace5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/db87f267-b523-4a65-ab4c-b343a2d9ace5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~projected/kube-api-access-9z9qh:{mountpoint:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~projected/kube-api-access-9z9qh major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c/volumes/kubernetes.io~projected/kube-api-access-cx2g5:{mountpoint:/var/lib/kubelet/pods/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c/volumes/kubernetes.io~projected/kube-api-access-cx2g5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f58038f2-47ce-43fc-bfa5-116eed7f4776/volumes/kubernetes.io~projected/kube-api-access-lj8dt:{mountpoint:/var/lib/kubelet/pods/f58038f2-47ce-43fc-bfa5-116eed7f4776/volumes/kubernetes.io~projected/kube-api-access-lj8dt major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5824bdb-ee34-4c21-8655-5016b9338e5c/volumes/kubernetes.io~projected/kube-api-access-9s8jw:{mountpoint:/var/lib/kubelet/pods/f5824bdb-ee34-4c21-8655-5016b9338e5c/volumes/kubernetes.io~projected/kube-api-access-9s8jw major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5824bdb-ee34-4c21-8655-5016b9338e5c/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/f5824bdb-ee34-4c21-8655-5016b9338e5c/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f67e5454-86f4-4e80-880a-070e4e7a38c1/volumes/kubernetes.io~projected/kube-api-access-wr8vg:{mountpoint:/var/lib/kubelet/pods/f67e5454-86f4-4e80-880a-070e4e7a38c1/volumes/kubernetes.io~projected/kube-api-access-wr8vg major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f67e5454-86f4-4e80-880a-070e4e7a38c1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f67e5454-86f4-4e80-880a-070e4e7a38c1/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f957aba3-d3ed-48a0-aff7-2c01130e95ee/volumes/kubernetes.io~projected/kube-api-access-8xc4q:{mountpoint:/var/lib/kubelet/pods/f957aba3-d3ed-48a0-aff7-2c01130e95ee/volumes/kubernetes.io~projected/kube-api-access-8xc4q major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f957aba3-d3ed-48a0-aff7-2c01130e95ee/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f957aba3-d3ed-48a0-aff7-2c01130e95ee/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d2f39ece51e21360f3a442f560bb21b6930240e5085c653db79b55a85bde4b74/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0a2c496d36ba7fc66ea1c71d01ebae14117ffc55ac1e88767bbcf77465f29ee6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4c0840df0a17b4024a93951c853525baebbb8bb3e1abfc205cef01d949f9c27f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/84e1eac302721e01d119862a46cbee9f556e9ecaccfd139158c09ddf7aeeaaca/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/88f0f8bc1fb194b7d82c8c651c0e2de9b7405ef9fee30b7351e373e8bb8c6078/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/76cdbb5f4d2abc695adf9c4983e0249b6303965740bbb170ae518e97a64e76db/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1872969336a764fbe32d4cca8ed6880ea0a95cc6996f4511a8d97971fe67e0d4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/974ab2a53e4611ef9901c8491b537324af93de155a3b89e86cdd3224d7764199/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/25f5f8c8e41f743dec2b4a30c632d43ec4f8be8dedb9a0b8ff8a844634bf416c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/9a9679f3f0f463bf6adc987a33ce0c8d0c3770d0dba7db45541fd2c9c320db3a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/452c6952acc9cc99d96ec43f022beb72b2ad851101923df6445864050636e08d/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/e290c29a2b607bb2b9843e5f27d9ac4893ec902fa00d5b59524f1dabefe7e015/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e81a4ca4c66a891b47368281f86a7b4d47dda4ad0a9d97eb1e99a9768defea6f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3d5804af2e7f917d920c81c7cf680cec41968968873946c837aeeecc6465fa01/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b18e8dfd92da27b0d6951fa3e4aed7437868928f0dc8ed14a1940fcd4deadaff/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/61368f152a793613080eedb385e8e45c179c31267643ef8cca7086d23ef86b0c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2c8e1361fa7e2f3c93051159ed6700ad6423c87846f3d487a09fd26258acb33d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cf35c848e05177f42521913b7a78ac80ca741b5978322e6194cfd31a2f8e5ece/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/78e03880f0fa3e45c638f74e35cf287212563f0a3685975315bc0499b4887bda/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/34b93a024ed72fe6d93447542b44aac36c4c5f44680588724e0d6ec61984fea4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b6d83f14b4663c2fe81eaeedc53777da1c6838f6b0a91c1e11fe918beef5b0ba/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/b9d9d89e7d38d5fd532a45c3dff31f7ac97ec761c71865f5b4ef1d3d614c907c/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/991afc59b5a74e20778f81d70ac3ea2e6d3c74964d45c8199014656bfcdc3fdd/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/654b35d8deeb4616a2d875465ed4d7c5876e9ef960c5c5f16d55fc59c6a47f80/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3403ed4ca260d554f29bade6e64606d3a6f6bb5f4e9b2a4ddd446b8601ea929d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/60575d887ba2791f6e7c62daa819a5ab6a1bd44386fb15d9c691fd50b0bc0558/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8e952a721bafe7834e997595f50d6380059ed990998b08db5983ece411054a33/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ef379724e6c82fc934752b3c97c1a6c9dc6ca9d670e0e3292202064a61ea5d12/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d532b58b81b662e83aaf29928583ff0da00ab491f82f1e8619763695e2264afb/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5530a8079a7ab2428dde5d6b3356c23788e1f47640f88d6b6f9035b2ae850009/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/a39d6472ee0937f806ad6152831eef7848ba1e0cf97bbae409154afd124a12c9/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/d4967e6570e88c752ccff7b9750e09bd2dda6026a3919d53a2d9c9d33145458e/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/26e6eb1cc7602e05c79ea480c8836179987ef917c4854a0c1ebf3a0eb4ad0fda/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/530461ffbcca3c1260e13ffad5bef370d462930ea01e051f78d44ae2b639b92b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/cfec935fb8664e26beeb8a1f11cd11aef3dc829367bab5e11dcd42bccb4ed9aa/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/0a046d108ce02751ee0b4f3523e47fa033601552fd09f16167bf9fa3f5052c9f/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/183b5ecd53820f99473b84b121641fb0edd7944ee06d84d88030f62ac3fa80de/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/1a5cd9ca1c483f6a62e236f7de0f0434d65ebcab32c18fed7be8b3897cb73250/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/41be101cacb6c03ef5e7d81fec65f37384caf81162072a4675e64179c86d7269/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/8a9e5c3f6e679d34dc53a9871ab74c4175ab63fcb4227a35c5ffc3ebf45a51ee/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3fd276ff809e9a3aba2d418f4798e4adc505548e0d4cb2f69bdbaac38ed17911/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/c117786691ec4d3e014aca6080b2bec6fc73facda4589df2278967f644c932c1/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/9d058e1f8b3ab2303e10c6a534551abad9535e3b37da1e7cbfe9c98341d48708/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/fa3d8826c54eca476238177227f1628c7cad2e9297138de5c2357a75f04598f5/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/dac916686ebb634705a93bc20028dca9b93e293075b950ea86aba3d4851d255a/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/abde9b02b2ca27d61596b784eb33cda2bf59f230bd9528bd529a53db67df1373/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/331ea2199f5d68f003b39fb0010dca7483a8a05f862442730f0fa25d1fce54c8/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/68174c120a812210ddb7ce8322adfeb4ef7f1590d64a9b89b1c5eaa882c6c39d/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/c36e3e999b45b67e804e5b935c9901a9d35c724bdba4c16261d0e0d8dfdeb82d/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/c9989a2fc13db44d001708b3af1cf4dab074b2fe75a031938a91b26bdf8bf77e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/80dad8d4dfba68b9d220e800554231329aa7dcf08ad4b013444344308422c1e9/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/292412de93cd75a0d2cb881f65be77aff48b478dfb5f36b3cb09b5cde8c161ef/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/e0105a83c203e82a5c25a6462d6493020dd07998e41bb1d1f100011e6049e30c/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/505d6be3e6ddb6f31512536baf423de23ee87ef7dd2f16d0c42cc9466d67aa3e/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/4c37cfe508823bf2196cf82daa53cd5aa6ae51bc541d7747b9a7e2f6b74e8c30/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/f31da98aa174c962432c0df9a437938ecb32cdbcb328db93547ad3986fcb10ad/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/33fa1502947a7a33a7c6e0215c9872b68c4f2bc3e73c72020a9ad3992e782f2f/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/0275899e35a983995f5cc2c5b5544f9f8fe4497a006d6ea889986ef6caa04491/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/dd0e47cb912813e7e3e291f6c1538facc7a099d7dec40e7fd22107b70caa8891/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/b3ece3543c8c191bacca4744ab57b23ad880370ed58cafe53a69c39bf7695492/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/d26caddf31fb51819045e3faa697828f2376862a87f5748f4f0649bd49f16a7a/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/cea49702b8b892996be05ebb22b3a85cc2913807beac9a841d2379438d1977a9/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/60d25fc402f3427dbefbfed9c5e400fac7dfd2ed595c667f7db8b67136af121a/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/bf1f7ea44d615bb893deeb4869d3285b1fe76601e4144e6cd7d192b6b08c8e01/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/cc85fa8d08173a2272208ac603f0b5d1b898775d7e39fe9b3f5b3bd8a309ce1a/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/7dcdf90d46421ba56062516b2c8bbdb5268fdc0ebd7a54860ecac724e6bbc109/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4c35952ddbfdaca79982ba83a17061b9d573826dbf1ceb15c25d833a0c2e0108/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/12feaab73eb4efdfb7da67a58bf740e9c1fcafb16f0896f99e5711c0cf86d2e8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/868e0f2666487033fe7bc9b87ea288fd28d4c915d6e1ad774e219ad799e31542/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f95832f2445b3f879e0e81c39111c7997f8ee71a859e69626dd8064cb9cb00c6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/94688b13894adcccc1134bfa331985f54ed2ec967dd87e69d3031a443e59cbcb/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/27d1e80f8d348374d9d21ba0595da554ffebf8c8f594d4c96dc042b3f3ede7c4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/08f669eb3320a011c0286821ce1d63d7302f07d0883f3db47743995c9e7f8678/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8003f619f071fe4160d4ce21e9a7844d49353d9b91e5efbbf314c1a47d0ea5fd/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8e70367f764f33a155375515aca47560bd0efbd39c71b541b696d08bb6e98dcc/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6ff408d6ac7b2d4d1be0ed3ad73058acf376ed56246ce66fa47501d2b72b14e1/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 18:21:19.491214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.490759 5688 manager.go:217] Machine: {Timestamp:2026-10-31 18:21:19.489951203 +0000 UTC m=+0.092092339 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200084 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2a141c5a0550a5fee13bfa60519c SystemUUID:ec2d2a14-1c5a-0550-a5fe-e13bfa60519c BootID:5d45d1a3-b11e-4f82-9136-d3314b7fd47c Filesystems:[{Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c/volumes/kubernetes.io~projected/kube-api-access-cx2g5 DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f3243dabd91b7158ba3331478d06a86efb6fb78c9bd49024925ba5b315def69/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bad584f95d6a1412fa623f8726550f8b0a3f0d05b1d2ef62624faefbcf84a1c1/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a45983b7fe0d729451cc8031478d99f97fa33f1906243c73b27888d5c4cc72ef/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f67e5454-86f4-4e80-880a-070e4e7a38c1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1090ee264181163f3bbaae8c25570adf2bb68f3b3dce42fcf987091b4efacc78/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f5824bdb-ee34-4c21-8655-5016b9338e5c/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:207 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3cc3b19bdc8ae69614f74ad1db5446a9aa604115f69c07c00e847d6f02c2ba9/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12259580-c199-4cac-a5c2-35a79e28ca90/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/kube-api-access-k2vvm DeviceMajor:0 DeviceMinor:219 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f67e5454-86f4-4e80-880a-070e4e7a38c1/volumes/kubernetes.io~projected/kube-api-access-wr8vg DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/848450994cd479a7515c15632266ea4ed84436792c04172803cf60e3d14d524c/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f58038f2-47ce-43fc-bfa5-116eed7f4776/volumes/kubernetes.io~projected/kube-api-access-lj8dt DeviceMajor:0 DeviceMinor:221 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f957aba3-d3ed-48a0-aff7-2c01130e95ee/volumes/kubernetes.io~projected/kube-api-access-8xc4q DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5db983a9ea25b0fc99460b3502f005001f49ddc135c104934610486b5e8a2766/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c829a43ef47a24a5b2038a49d7f03484554e5615b5726dc1a84a68914ccf943/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9a14654c8d5d8fd6e6f5dd53990c6d40d65823da1dd51c4c984dd98888a550d/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c9621db0-ac02-43c2-9edd-ec7bd05d1817/volumes/kubernetes.io~projected/kube-api-access-qj7t7 DeviceMajor:0 DeviceMinor:361 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/52aa29ef-34be-481f-b3a6-d78bfeb0b181/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d33e27-7f70-4b3c-b90d-927c598a0630/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/52aa29ef-34be-481f-b3a6-d78bfeb0b181/volumes/kubernetes.io~projected/kube-api-access-dgj2q DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b199eed4ea364e9c48a6cbddaae81f0091336bb9a8e05b8754f87f5ee544a59/userdata/shm DeviceMajor:0 DeviceMinor:394 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~projected/kube-api-access-9z9qh DeviceMajor:0 DeviceMinor:210 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f957aba3-d3ed-48a0-aff7-2c01130e95ee/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f6b2a0e97774a351b37c744c147b5065905e96c9152a3f6163c5a057b8563a0/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25825c7b-a2c6-4263-bc62-d2c85c00ec47/volumes/kubernetes.io~projected/kube-api-access-tww9t DeviceMajor:0 DeviceMinor:211 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0969d74c8efdda5166ca1940fe8a511574b34c7e2876a8bfc6810b228f48988a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a9778c230d77be0c6a477517a4fa4cd3b913455c7a8dfffeac8a3f50f92e04c/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3803e3109af7c2104f99be5bf4179a37fb9352ddd3fa14df12172e19a08a24b4/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3e4c1cca933c3e601d8a033c0f83fb798088fa744e40de3afc715236b51a40a/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/25de07e5-55c8-4c35-96af-7861b0a74469/volumes/kubernetes.io~projected/kube-api-access-ltprw DeviceMajor:0 DeviceMinor:293 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759/volumes/kubernetes.io~projected/kube-api-access-xh4nj DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12259580-c199-4cac-a5c2-35a79e28ca90/volumes/kubernetes.io~projected/kube-api-access-89b5x DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~projected/kube-api-access-wbr4h DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cc74a8fe-8053-4b78-b8bf-569e5e34a89f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/875a3d19-4e6c-4bb6-87b9-e801cd4f6604/volumes/kubernetes.io~projected/kube-api-access-ggrkf DeviceMajor:0 DeviceMinor:205 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d5c4caa285c8ad2939ac99ce73562650cb464fb4046f095f3d54efbd6804dd9/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aef0cda84ca45c834d349f52205c388e6c9d7e9fdb7a52317bf89ef836fc63a7/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db87f267-b523-4a65-ab4c-b343a2d9ace5/volumes/kubernetes.io~projected/kube-api-access-hd6kt DeviceMajor:0 DeviceMinor:233 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/96367807-1ef3-4c30-b605-d041d6f53ae8/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:347 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:392 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97b0ed79737662ce2076796474ecb2f15e48522038914f36bae46d82e72a9610/userdata/shm DeviceMajor:0 DeviceMinor:396 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/995393551e8f7cef944fb7b6163efb88d19791f9aa2b03c08891b9849dbd6e8d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb2184c5c8477cd968ccb50ee3d0f5e09e5ab9e4fcca78168e24ad7b0f484fc7/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/512c0bfc-c41f-4abd-b4cd-c85d358fac3e/volumes/kubernetes.io~projected/kube-api-access-n4qp6 DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/89f9210d-bf91-47c5-a4ab-318f8da1985d/volumes/kubernetes.io~projected/kube-api-access-nnd79 DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0906409-dea7-4779-9d86-4c9bfee0ee91/volumes/kubernetes.io~projected/kube-api-access-cksmw DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db87f267-b523-4a65-ab4c-b343a2d9ace5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:330 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25de07e5-55c8-4c35-96af-7861b0a74469/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:438 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3465fba7973d1dffdf612335059ffd6215d3e7b837fde9dc6b34ffca3c9a1ccf/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96e03cbb-2909-4802-accd-002dd2922c42/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:393 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/474a81eb-340b-45fe-8033-0deaecf35f03/volumes/kubernetes.io~projected/kube-api-access-csg5w DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/00096998-7fc2-406c-b705-c4f80a552391/volumes/kubernetes.io~projected/kube-api-access-bp6xm DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5824bdb-ee34-4c21-8655-5016b9338e5c/volumes/kubernetes.io~projected/kube-api-access-9s8jw DeviceMajor:0 DeviceMinor:224 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2da5603647dd25425daee5a88755a7a281087a098101bbb80ffcf1307d6dc717/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d33e27-7f70-4b3c-b90d-927c598a0630/volumes/kubernetes.io~projected/kube-api-access-sclxl DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9d5010af-7674-4de3-ac32-4799caaf4f39/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85/volumes/kubernetes.io~projected/kube-api-access-4xpmx DeviceMajor:0 DeviceMinor:286 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a4869852bdf43e4b9ce86b9c9497ac2d7bcfa45c410d9db700a2bc531d79041/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d5010af-7674-4de3-ac32-4799caaf4f39/volumes/kubernetes.io~projected/kube-api-access-29596 DeviceMajor:0 DeviceMinor:232 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e84588424bedd70a7a8134ed40435ce4096259a491ec9a98764f8d42f0b9bf8/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/335db604-5a37-42ad-a5b6-4536a8f5da96/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4334dda9-cc0b-49bd-a5ca-48457df595fe/volumes/kubernetes.io~projected/kube-api-access-6wjtb DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f17f5606-10aa-4f02-a2c4-16c28aba8540/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bde2551472e79836ef0f5cbe9431e0bde4b4aef181a5562bf8f8c751bf3967cc/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/96818867-d92e-402c-b179-d93d211f45a1/volumes/kubernetes.io~projected/kube-api-access-kcp6k DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e336539e47496ab9d1539bb9354fe9bbe90e07ddf0665cea09179545cb10be8/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ea0ef1504ddf33677c1d7675b146837bffa2309205dac6ae0206428dd695122/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ce43312-6761-45de-8350-d347681cb539/volumes/kubernetes.io~projected/kube-api-access-5zn6x DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1725d7e84b5d324a69bbeef486876bccee83cacc046c94037e9aac2357ee79cf/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a406067bb9d2eab6d39b5cbe4af772fbce34162860bb30b7f1a3368b3ab627e7/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1090ee264181163 MacAddress:8a:32:e9:60:f5:57 Speed:10000 Mtu:8901} {Name:2da5603647dd254 MacAddress:26:23:00:0a:d3:92 Speed:10000 Mtu:8901} {Name:2ea0ef1504ddf33 MacAddress:56:13:c0:ee:3c:13 Speed:10000 Mtu:8901} {Name:3803e3109af7c21 MacAddress:62:a1:53:cf:c0:7c Speed:10000 Mtu:8901} {Name:4b199eed4ea364e MacAddress:06:1c:f6:a3:ac:08 Speed:10000 Mtu:8901} {Name:6a9778c230d77be MacAddress:22:06:98:0a:f3:a7 Speed:10000 Mtu:8901} {Name:7a4869852bdf43e MacAddress:5a:aa:bf:7b:10:61 Speed:10000 Mtu:8901} {Name:848450994cd479a MacAddress:d6:72:f5:5d:e7:0b Speed:10000 Mtu:8901} {Name:97b0ed79737662c MacAddress:e6:d4:b4:f1:3d:b9 Speed:10000 Mtu:8901} {Name:9d5c4caa285c8ad MacAddress:9a:5b:d3:93:f2:5d Speed:10000 Mtu:8901} {Name:9e336539e47496a MacAddress:32:60:80:73:c9:ad Speed:10000 Mtu:8901} {Name:9e84588424bedd7 MacAddress:8e:e7:62:1e:cc:f0 Speed:10000 Mtu:8901} {Name:a406067bb9d2eab MacAddress:f2:15:11:09:ab:12 Speed:10000 Mtu:8901} {Name:a45983b7fe0d729 MacAddress:fa:f9:e6:bb:62:eb Speed:10000 Mtu:8901} {Name:aef0cda84ca45c8 MacAddress:1e:0b:a2:37:eb:eb Speed:10000 Mtu:8901} {Name:b9a14654c8d5d8f MacAddress:f2:9b:16:0b:44:2d Speed:10000 Mtu:8901} {Name:bad584f95d6a141 MacAddress:4a:0c:0c:62:96:6e Speed:10000 Mtu:8901} {Name:bb2184c5c8477cd MacAddress:66:e7:94:82:6d:cd Speed:10000 Mtu:8901} {Name:bde2551472e7983 MacAddress:7e:19:c5:d8:11:94 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:44:ee:6e:d0:39 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:07:30:99:9d:2c Speed:0 Mtu:8901} {Name:c3cc3b19bdc8ae6 MacAddress:fa:54:d9:07:d7:68 Speed:10000 Mtu:8901} {Name:e3e4c1cca933c3e MacAddress:ce:48:b4:46:95:0c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:44:ee:6e:d0:39 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:36:07:84:dd:3a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:64:48:02:ce:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 18:21:19.491502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491221 5688 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 18:21:19.491502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491303 5688 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 18:21:19.491595 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491572 5688 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 18:21:19.491731 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491597 5688 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-47.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 18:21:19.491774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491734 5688 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 18:21:19.491774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491742 5688 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 18:21:19.491774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491752 5688 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 18:21:19.491774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491765 5688 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 18:21:19.491874 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491781 5688 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 18:21:19.491938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.491929 5688 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 18:21:19.492032 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492024 5688 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 18:21:19.492067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492059 5688 kubelet.go:493] "Attempting to sync node with API server" Oct 31 18:21:19.492093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492073 5688 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 18:21:19.492093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492083 5688 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 18:21:19.492093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492091 5688 kubelet.go:405] "Adding apiserver pod source" Oct 31 18:21:19.492218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492098 5688 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 18:21:19.492798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492781 5688 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 18:21:19.492798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.492793 5688 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 18:21:19.493265 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.493245 5688 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 18:21:19.506530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506513 5688 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 18:21:19.506530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506528 5688 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 18:21:19.506647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506618 5688 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 18:21:19.506800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506789 5688 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 18:21:19.506829 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506817 5688 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 18:21:19.506906 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506898 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 18:21:19.506935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506912 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 18:21:19.506935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506919 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 18:21:19.506935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506924 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 18:21:19.506935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506929 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 18:21:19.506935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.506935 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 18:21:19.507108 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507096 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 18:21:19.507162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507114 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 18:21:19.507162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507125 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 18:21:19.507162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507135 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 18:21:19.507162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507149 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 18:21:19.507351 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507165 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 18:21:19.507351 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507246 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 18:21:19.507351 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.507265 5688 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 18:21:19.511394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.511379 5688 server.go:1263] "Started kubelet" Oct 31 18:21:19.511520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.511479 5688 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 18:21:19.511833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.511772 5688 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 18:21:19.511904 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.511847 5688 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 18:21:19.511937 ip-10-0-140-47 systemd[1]: Started Kubernetes Kubelet. Oct 31 18:21:19.512148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.512112 5688 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 18:21:19.512762 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.512741 5688 server.go:317] "Adding debug handlers to kubelet server" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.516841 5688 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.516907 5688 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.517150 5688 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:08 +0000 UTC" deadline="2028-05-04 13:18:51.353315077 +0000 UTC" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.517230 5688 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13218h57m31.83608999s" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.518482 5688 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.518518 5688 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 18:21:19.518700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.518525 5688 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 18:21:19.519065 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.519002 5688 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 18:21:19.524886 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524849 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52aa29ef-34be-481f-b3a6-d78bfeb0b181" volumeName="kubernetes.io/configmap/52aa29ef-34be-481f-b3a6-d78bfeb0b181-telemetry-config" seLinuxMountContext="" Oct 31 18:21:19.524886 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524885 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db87f267-b523-4a65-ab4c-b343a2d9ace5" volumeName="kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524900 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f957aba3-d3ed-48a0-aff7-2c01130e95ee" volumeName="kubernetes.io/configmap/f957aba3-d3ed-48a0-aff7-2c01130e95ee-config" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524910 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52aa29ef-34be-481f-b3a6-d78bfeb0b181" volumeName="kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524918 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875a3d19-4e6c-4bb6-87b9-e801cd4f6604" volumeName="kubernetes.io/projected/875a3d19-4e6c-4bb6-87b9-e801cd4f6604-kube-api-access-ggrkf" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524929 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f67e5454-86f4-4e80-880a-070e4e7a38c1" volumeName="kubernetes.io/configmap/f67e5454-86f4-4e80-880a-070e4e7a38c1-config" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524938 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="474a81eb-340b-45fe-8033-0deaecf35f03" volumeName="kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-multus-daemon-config" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524952 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="512c0bfc-c41f-4abd-b4cd-c85d358fac3e" volumeName="kubernetes.io/projected/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kube-api-access-n4qp6" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524975 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f9210d-bf91-47c5-a4ab-318f8da1985d" volumeName="kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524987 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f957aba3-d3ed-48a0-aff7-2c01130e95ee" volumeName="kubernetes.io/projected/f957aba3-d3ed-48a0-aff7-2c01130e95ee-kube-api-access-8xc4q" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.524999 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525011 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12259580-c199-4cac-a5c2-35a79e28ca90" volumeName="kubernetes.io/secret/12259580-c199-4cac-a5c2-35a79e28ca90-serving-cert" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525023 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25825c7b-a2c6-4263-bc62-d2c85c00ec47" volumeName="kubernetes.io/projected/25825c7b-a2c6-4263-bc62-d2c85c00ec47-kube-api-access-tww9t" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525032 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f9210d-bf91-47c5-a4ab-318f8da1985d" volumeName="kubernetes.io/projected/89f9210d-bf91-47c5-a4ab-318f8da1985d-kube-api-access-nnd79" seLinuxMountContext="" Oct 31 18:21:19.525052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525049 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce43312-6761-45de-8350-d347681cb539" volumeName="kubernetes.io/empty-dir/8ce43312-6761-45de-8350-d347681cb539-tmp-dir" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525063 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f957aba3-d3ed-48a0-aff7-2c01130e95ee" volumeName="kubernetes.io/secret/f957aba3-d3ed-48a0-aff7-2c01130e95ee-serving-cert" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525081 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00096998-7fc2-406c-b705-c4f80a552391" volumeName="kubernetes.io/secret/00096998-7fc2-406c-b705-c4f80a552391-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525093 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525106 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335db604-5a37-42ad-a5b6-4536a8f5da96" volumeName="kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-tuned" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525120 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce43312-6761-45de-8350-d347681cb539" volumeName="kubernetes.io/projected/8ce43312-6761-45de-8350-d347681cb539-kube-api-access-5zn6x" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525128 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9621db0-ac02-43c2-9edd-ec7bd05d1817" volumeName="kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525141 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4334dda9-cc0b-49bd-a5ca-48457df595fe" volumeName="kubernetes.io/projected/4334dda9-cc0b-49bd-a5ca-48457df595fe-kube-api-access-6wjtb" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525150 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525159 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25de07e5-55c8-4c35-96af-7861b0a74469" volumeName="kubernetes.io/empty-dir/25de07e5-55c8-4c35-96af-7861b0a74469-data-volume" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525169 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335db604-5a37-42ad-a5b6-4536a8f5da96" volumeName="kubernetes.io/projected/335db604-5a37-42ad-a5b6-4536a8f5da96-kube-api-access-wbr4h" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525178 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96367807-1ef3-4c30-b605-d041d6f53ae8" volumeName="kubernetes.io/configmap/96367807-1ef3-4c30-b605-d041d6f53ae8-nginx-conf" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525212 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5010af-7674-4de3-ac32-4799caaf4f39" volumeName="kubernetes.io/projected/9d5010af-7674-4de3-ac32-4799caaf4f39-kube-api-access-29596" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525225 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d33e27-7f70-4b3c-b90d-927c598a0630" volumeName="kubernetes.io/secret/b6d33e27-7f70-4b3c-b90d-927c598a0630-serving-cert" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525279 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc74a8fe-8053-4b78-b8bf-569e5e34a89f" volumeName="kubernetes.io/configmap/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-konnectivity-ca" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525294 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0906409-dea7-4779-9d86-4c9bfee0ee91" volumeName="kubernetes.io/projected/d0906409-dea7-4779-9d86-4c9bfee0ee91-kube-api-access-cksmw" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525305 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="474a81eb-340b-45fe-8033-0deaecf35f03" volumeName="kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-cni-binary-copy" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525314 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5010af-7674-4de3-ac32-4799caaf4f39" volumeName="kubernetes.io/secret/9d5010af-7674-4de3-ac32-4799caaf4f39-cert" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525324 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d33e27-7f70-4b3c-b90d-927c598a0630" volumeName="kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-tmp" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525335 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f17f5606-10aa-4f02-a2c4-16c28aba8540" volumeName="kubernetes.io/projected/f17f5606-10aa-4f02-a2c4-16c28aba8540-kube-api-access-9z9qh" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525352 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f67e5454-86f4-4e80-880a-070e4e7a38c1" volumeName="kubernetes.io/projected/f67e5454-86f4-4e80-880a-070e4e7a38c1-kube-api-access-wr8vg" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525371 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00096998-7fc2-406c-b705-c4f80a552391" volumeName="kubernetes.io/projected/00096998-7fc2-406c-b705-c4f80a552391-kube-api-access-bp6xm" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525385 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525402 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96818867-d92e-402c-b179-d93d211f45a1" volumeName="kubernetes.io/projected/96818867-d92e-402c-b179-d93d211f45a1-kube-api-access-kcp6k" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525414 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e03cbb-2909-4802-accd-002dd2922c42" volumeName="kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525427 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f17f5606-10aa-4f02-a2c4-16c28aba8540" volumeName="kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-default-certificate" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525443 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f17f5606-10aa-4f02-a2c4-16c28aba8540" volumeName="kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-stats-auth" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525456 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d33e27-7f70-4b3c-b90d-927c598a0630" volumeName="kubernetes.io/projected/b6d33e27-7f70-4b3c-b90d-927c598a0630-kube-api-access-sclxl" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525472 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00096998-7fc2-406c-b705-c4f80a552391" volumeName="kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-script-lib" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525486 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525498 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12259580-c199-4cac-a5c2-35a79e28ca90" volumeName="kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-config" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525515 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25de07e5-55c8-4c35-96af-7861b0a74469" volumeName="kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525528 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335db604-5a37-42ad-a5b6-4536a8f5da96" volumeName="kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-tmp" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525546 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52aa29ef-34be-481f-b3a6-d78bfeb0b181" volumeName="kubernetes.io/projected/52aa29ef-34be-481f-b3a6-d78bfeb0b181-kube-api-access-dgj2q" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525559 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759" volumeName="kubernetes.io/projected/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-kube-api-access-xh4nj" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525573 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00096998-7fc2-406c-b705-c4f80a552391" volumeName="kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-config" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525630 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25de07e5-55c8-4c35-96af-7861b0a74469" volumeName="kubernetes.io/projected/25de07e5-55c8-4c35-96af-7861b0a74469-kube-api-access-ltprw" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525738 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759" volumeName="kubernetes.io/configmap/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-iptables-alerter-script" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525776 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f9210d-bf91-47c5-a4ab-318f8da1985d" volumeName="kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-binary-copy" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525788 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0906409-dea7-4779-9d86-4c9bfee0ee91" volumeName="kubernetes.io/configmap/d0906409-dea7-4779-9d86-4c9bfee0ee91-serviceca" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525800 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db87f267-b523-4a65-ab4c-b343a2d9ace5" volumeName="kubernetes.io/empty-dir/db87f267-b523-4a65-ab4c-b343a2d9ace5-tmp-dir" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525816 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db87f267-b523-4a65-ab4c-b343a2d9ace5" volumeName="kubernetes.io/projected/db87f267-b523-4a65-ab4c-b343a2d9ace5-kube-api-access-hd6kt" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525851 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00096998-7fc2-406c-b705-c4f80a552391" volumeName="kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-env-overrides" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525871 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525885 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525898 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25de07e5-55c8-4c35-96af-7861b0a74469" volumeName="kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525915 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfe1fd5-4835-4b2b-87d6-d218cfcd6b85" volumeName="kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525928 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f17f5606-10aa-4f02-a2c4-16c28aba8540" volumeName="kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525946 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5824bdb-ee34-4c21-8655-5016b9338e5c" volumeName="kubernetes.io/configmap/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-config" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525949 5688 factory.go:55] Registering systemd factory Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525962 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12259580-c199-4cac-a5c2-35a79e28ca90" volumeName="kubernetes.io/projected/12259580-c199-4cac-a5c2-35a79e28ca90-kube-api-access-89b5x" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525977 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f9210d-bf91-47c5-a4ab-318f8da1985d" volumeName="kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 18:21:19.525969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525997 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d33e27-7f70-4b3c-b90d-927c598a0630" volumeName="kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-trusted-ca-bundle" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526011 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d33e27-7f70-4b3c-b90d-927c598a0630" volumeName="kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-snapshots" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526027 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" volumeName="kubernetes.io/projected/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-kube-api-access-cx2g5" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526041 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5824bdb-ee34-4c21-8655-5016b9338e5c" volumeName="kubernetes.io/projected/f5824bdb-ee34-4c21-8655-5016b9338e5c-kube-api-access-9s8jw" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526054 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" volumeName="kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526073 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="474a81eb-340b-45fe-8033-0deaecf35f03" volumeName="kubernetes.io/projected/474a81eb-340b-45fe-8033-0deaecf35f03-kube-api-access-csg5w" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526100 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f17f5606-10aa-4f02-a2c4-16c28aba8540" volumeName="kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526117 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f22f6aaa-22cc-4ab9-a4bb-90d73eac861c" volumeName="kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526132 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58038f2-47ce-43fc-bfa5-116eed7f4776" volumeName="kubernetes.io/projected/f58038f2-47ce-43fc-bfa5-116eed7f4776-kube-api-access-lj8dt" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526145 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96367807-1ef3-4c30-b605-d041d6f53ae8" volumeName="kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526164 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d33e27-7f70-4b3c-b90d-927c598a0630" volumeName="kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-service-ca-bundle" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526178 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc74a8fe-8053-4b78-b8bf-569e5e34a89f" volumeName="kubernetes.io/secret/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-agent-certs" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526215 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db87f267-b523-4a65-ab4c-b343a2d9ace5" volumeName="kubernetes.io/configmap/db87f267-b523-4a65-ab4c-b343a2d9ace5-config-volume" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526230 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5824bdb-ee34-4c21-8655-5016b9338e5c" volumeName="kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526244 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f67e5454-86f4-4e80-880a-070e4e7a38c1" volumeName="kubernetes.io/secret/f67e5454-86f4-4e80-880a-070e4e7a38c1-serving-cert" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526263 5688 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12259580-c199-4cac-a5c2-35a79e28ca90" volumeName="kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-trusted-ca" seLinuxMountContext="" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526275 5688 reconstruct.go:97] "Volume reconstruction finished" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526283 5688 reconciler.go:29] "Reconciler: start to sync state" Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.525966 5688 factory.go:223] Registration of the systemd container factory successfully Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526735 5688 factory.go:153] Registering CRI-O factory Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526757 5688 factory.go:223] Registration of the crio container factory successfully Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526813 5688 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526866 5688 factory.go:103] Registering Raw factory Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.526886 5688 manager.go:1199] Started watching for new ooms in manager Oct 31 18:21:19.529204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.527387 5688 manager.go:319] Starting recovery of all containers Oct 31 18:21:19.539708 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.539676 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:21:19.540404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.540379 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:21:19.549361 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.549064 5688 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 18:21:19.558375 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.557976 5688 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 18:21:19.558487 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.558383 5688 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 18:21:19.558487 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.558407 5688 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 18:21:19.558487 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.558450 5688 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 18:21:19.603705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.603626 5688 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="9a17508a36c2935a03ec7d802abb7b202d3ea6d2f9ab640c18cc88658b8673f6" exitCode=0 Oct 31 18:21:19.603705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.603652 5688 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="16e3463c79cb2e4d1c833024ebb8f21c801676d2fc907972a9034f74b42a820b" exitCode=0 Oct 31 18:21:19.603705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.603660 5688 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="ad5e1f336d83c1502f583a80d8a2d167cce2068193a0c4c22e3025026883c845" exitCode=0 Oct 31 18:21:19.603705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.603668 5688 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="6c8a554dbccad95135c3fa9a5ce9ac0c2eceb6f860495e4026413980faf3f935" exitCode=0 Oct 31 18:21:19.603705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.603677 5688 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="441d1d525a441b28f251fd2fd64267261bc75c0e6d8c87b1e353b2d33645a194" exitCode=0 Oct 31 18:21:19.603705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.603685 5688 generic.go:356] "Generic (PLEG): container finished" podID="89f9210d-bf91-47c5-a4ab-318f8da1985d" containerID="ca14fc0f427d617e21e1596315bdb3b21c92086bca8338a1cc46c596104de607" exitCode=0 Oct 31 18:21:19.614399 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.614373 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:21:19.614520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.614409 5688 generic.go:356] "Generic (PLEG): container finished" podID="12259580-c199-4cac-a5c2-35a79e28ca90" containerID="ca740c27ab199d9f4f1113e32e9acd6fa6571ec20bd2a9aabec12fb15a5fe80c" exitCode=255 Oct 31 18:21:19.618549 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.618524 5688 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:21:19.630891 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.630866 5688 generic.go:356] "Generic (PLEG): container finished" podID="9182be3e97d0133207671d4d8f6ed1f3" containerID="0b51e91abd07adbc6805190e7bc2ce8a4d6567ec4920d250d01deb5fd4d68cf4" exitCode=0 Oct 31 18:21:19.651565 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.651546 5688 manager.go:324] Recovery completed Oct 31 18:21:19.658543 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.658524 5688 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 18:21:19.671613 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.671592 5688 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:21:19.673739 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.673722 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:21:19.673825 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.673752 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:21:19.673825 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.673764 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:21:19.676090 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676074 5688 cpu_manager.go:226] "Starting" policy="none" Oct 31 18:21:19.676157 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676089 5688 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:21:19.676157 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676090 5688 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 18:21:19.676225 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676163 5688 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 18:21:19.676316 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676302 5688 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 18:21:19.676346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676316 5688 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 18:21:19.676346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676335 5688 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 18:21:19.676346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676340 5688 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 18:21:19.676346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676346 5688 policy_none.go:50] "Start" Oct 31 18:21:19.676457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676356 5688 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 18:21:19.676457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676365 5688 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 18:21:19.676512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676490 5688 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 18:21:19.676512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676497 5688 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 18:21:19.676512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.676502 5688 policy_none.go:44] "Start" Oct 31 18:21:19.679492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.677661 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:21:19.679492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.677696 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:21:19.679492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.677712 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:21:19.682821 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.682789 5688 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 18:21:19.685463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.685440 5688 manager.go:341] "Starting Device Plugin manager" Oct 31 18:21:19.685547 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.685471 5688 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 18:21:19.685547 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.685497 5688 server.go:87] "Starting device plugin registration server" Oct 31 18:21:19.685823 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.685809 5688 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 18:21:19.685878 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.685825 5688 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 18:21:19.685935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.685916 5688 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 18:21:19.686051 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.686020 5688 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 18:21:19.686102 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.686058 5688 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 18:21:19.686102 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.686063 5688 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 18:21:19.686964 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.686887 5688 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T18:21:19.686049031Z","UUID":"5d566f7e-e63f-4337-9539-a51424ea75d5","Handler":null,"Name":"","Endpoint":""} Oct 31 18:21:19.687907 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.687842 5688 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 18:21:19.687907 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.687881 5688 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-47.us-west-1.compute.internal\" not found" Oct 31 18:21:19.688317 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.688296 5688 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 18:21:19.688411 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.688323 5688 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 18:21:19.699334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.699312 5688 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:21:19.699334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.699331 5688 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:21:19.699491 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.699424 5688 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 18:21:19.728023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.728002 5688 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:21:19.777758 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.777741 5688 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 18:21:19.786948 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.786921 5688 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 18:21:19.788074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.788055 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 18:21:19.788247 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.788091 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 18:21:19.788247 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.788107 5688 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 18:21:19.788247 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.788205 5688 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.858794 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.858711 5688 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal"] Oct 31 18:21:19.859400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859366 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"4d60099914d5eef84a51e7038f8e20de","Type":"ContainerStarted","Data":"0d5fb4755a93f8fddc12667b5ac35cae974d8698f4e55152dae9d463d82a5bc4"} Oct 31 18:21:19.859501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859407 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"4d60099914d5eef84a51e7038f8e20de","Type":"ContainerStarted","Data":"aa86bc7b78431c54115b4c257d534eda53d71a350ebf163cdf007fa644934a72"} Oct 31 18:21:19.859501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859434 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"9182be3e97d0133207671d4d8f6ed1f3","Type":"ContainerStarted","Data":"62fec414e7240bf11b8f4ab16935a48dee9792e0b9176dfb7508651c6c765d4e"} Oct 31 18:21:19.859501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859447 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"9182be3e97d0133207671d4d8f6ed1f3","Type":"ContainerDied","Data":"0b51e91abd07adbc6805190e7bc2ce8a4d6567ec4920d250d01deb5fd4d68cf4"} Oct 31 18:21:19.859501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859455 5688 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.859655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859456 5688 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.859655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.859459 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" event={"ID":"9182be3e97d0133207671d4d8f6ed1f3","Type":"ContainerStarted","Data":"f40cb75437470d05139934ecfbf451ccaeb39144c0e2394ebf714a30c001a475"} Oct 31 18:21:19.900810 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.900787 5688 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.900923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.900840 5688 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.923602 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.923583 5688 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 18:21:19.923694 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.923633 5688 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.924416 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.924403 5688 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 18:21:19.924487 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:19.924431 5688 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.999802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.999773 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.999952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.999809 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:19.999952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:19.999837 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d60099914d5eef84a51e7038f8e20de-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"4d60099914d5eef84a51e7038f8e20de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.100300 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.100268 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.100300 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.100302 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.100479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.100320 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d60099914d5eef84a51e7038f8e20de-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"4d60099914d5eef84a51e7038f8e20de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.100479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.100361 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.100479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.100368 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9182be3e97d0133207671d4d8f6ed1f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"9182be3e97d0133207671d4d8f6ed1f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.100479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.100406 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d60099914d5eef84a51e7038f8e20de-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" (UID: \"4d60099914d5eef84a51e7038f8e20de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.492554 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.492526 5688 apiserver.go:52] "Watching apiserver" Oct 31 18:21:20.597828 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.597799 5688 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 18:21:20.599803 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.599766 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xncwq","kube-system/kas-connection-checker-685895c7c9-ksnkf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t","kube-system/kas-connection-checker-685895c7c9-xgpld","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c","openshift-ovn-kubernetes/ovnkube-node-pbjfs","openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz","kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-tczvl","openshift-network-operator/iptables-alerter-rcrmh","openshift-console-operator/console-operator-7cd56b8dbc-zqcnd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm","openshift-image-registry/node-ca-hf5q2","openshift-ingress-canary/ingress-canary-px79l","openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2","openshift-image-registry/image-registry-85cbfd9d9f-q2vwz","openshift-ingress/router-default-59bdcd5559-qvgdn","openshift-cluster-node-tuning-operator/tuned-sklpw","kube-system/konnectivity-agent-v2bs5","openshift-dns/dns-default-pcvwd","openshift-insights/insights-operator-7c4986d449-8z6s5","openshift-dns/node-resolver-z89h2","openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v","openshift-multus/multus-pz8h8","openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz","openshift-network-diagnostics/network-check-target-bch5w","openshift-multus/network-metrics-daemon-xkth6","openshift-multus/multus-additional-cni-plugins-g94x2","kube-system/kas-connection-checker-685895c7c9-v4vbz","openshift-insights/insights-runtime-extractor-np8hr"] Oct 31 18:21:20.621390 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.621368 5688 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 18:21:20.647289 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.647264 5688 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.647404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.647339 5688 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.653669 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.653654 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.654368 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.654351 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.654368 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.654353 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.654615 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.654602 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.655684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.655670 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 18:21:20.655743 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.655671 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-whbd6\"" Oct 31 18:21:20.655883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.655866 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6fctz\"" Oct 31 18:21:20.655937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.655893 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 18:21:20.655937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.655929 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7pnqt\"" Oct 31 18:21:20.656220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656205 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 18:21:20.656323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656253 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 18:21:20.656362 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656266 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zxchx\"" Oct 31 18:21:20.656508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656491 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.656584 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656496 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.656584 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656508 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.656724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656705 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6pq8w\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656753 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656766 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656779 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656790 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656780 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656752 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9kc2n\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656787 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656784 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656798 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656812 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.656920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656831 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 18:21:20.657612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656937 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.657612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.656978 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.657612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.657304 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 18:21:20.657853 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.657835 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 18:21:20.702455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.702434 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-52br2\"" Oct 31 18:21:20.703372 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703355 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ce43312-6761-45de-8350-d347681cb539-hosts-file\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.703456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703382 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-script-lib\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.703456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703399 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.703456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703420 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-default-certificate\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.703456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703446 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9qh\" (UniqueName: \"kubernetes.io/projected/f17f5606-10aa-4f02-a2c4-16c28aba8540-kube-api-access-9z9qh\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.703630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703479 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 18:21:20.703630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703502 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.703630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703523 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\"" Oct 31 18:21:20.703860 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703839 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.703895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703871 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-socket-dir-parent\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.703895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703890 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.703957 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703913 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8dt\" (UniqueName: \"kubernetes.io/projected/f58038f2-47ce-43fc-bfa5-116eed7f4776-kube-api-access-lj8dt\") pod \"volume-data-source-validator-58c9f5f46d-98g5c\" (UID: \"f58038f2-47ce-43fc-bfa5-116eed7f4776\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:21:20.703990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.703955 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-run\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.704037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704023 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-cni-binary-copy\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.704118 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704103 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-k8s-cni-cncf-io\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.704179 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704138 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-netns\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.704179 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704161 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-system-cni-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.704275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704224 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.704275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704254 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:20.704374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704276 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:20.704374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704315 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-modprobe-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.704374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704349 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d33e27-7f70-4b3c-b90d-927c598a0630-serving-cert\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.704505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704383 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.704505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704401 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-host\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.704505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704419 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.704505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704435 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjtb\" (UniqueName: \"kubernetes.io/projected/4334dda9-cc0b-49bd-a5ca-48457df595fe-kube-api-access-6wjtb\") pod \"kas-connection-checker-685895c7c9-ksnkf\" (UID: \"4334dda9-cc0b-49bd-a5ca-48457df595fe\") " pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:21:20.704505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704482 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52aa29ef-34be-481f-b3a6-d78bfeb0b181-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:20.704505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704486 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-multus-certs\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704543 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704579 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29596\" (UniqueName: \"kubernetes.io/projected/9d5010af-7674-4de3-ac32-4799caaf4f39-kube-api-access-29596\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704612 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrkf\" (UniqueName: \"kubernetes.io/projected/875a3d19-4e6c-4bb6-87b9-e801cd4f6604-kube-api-access-ggrkf\") pod \"kas-connection-checker-685895c7c9-xgpld\" (UID: \"875a3d19-4e6c-4bb6-87b9-e801cd4f6604\") " pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704640 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-host-slash\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704668 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-kubelet\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704690 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-hostroot\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.704759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704735 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-config\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704779 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qp6\" (UniqueName: \"kubernetes.io/projected/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kube-api-access-n4qp6\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704799 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f957aba3-d3ed-48a0-aff7-2c01130e95ee-config\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704822 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704858 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704872 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-log-socket\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704888 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-netd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704917 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6xm\" (UniqueName: \"kubernetes.io/projected/00096998-7fc2-406c-b705-c4f80a552391-kube-api-access-bp6xm\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704955 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25de07e5-55c8-4c35-96af-7861b0a74469-crio-socket\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704963 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f957aba3-d3ed-48a0-aff7-2c01130e95ee-config\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704964 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-config\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.705069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.704992 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705099 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd79\" (UniqueName: \"kubernetes.io/projected/89f9210d-bf91-47c5-a4ab-318f8da1985d-kube-api-access-nnd79\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705136 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f67e5454-86f4-4e80-880a-070e4e7a38c1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705223 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87f267-b523-4a65-ab4c-b343a2d9ace5-tmp-dir\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705327 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f957aba3-d3ed-48a0-aff7-2c01130e95ee-serving-cert\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705354 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f67e5454-86f4-4e80-880a-070e4e7a38c1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705364 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87f267-b523-4a65-ab4c-b343a2d9ace5-tmp-dir\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.705422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705366 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbr4h\" (UniqueName: \"kubernetes.io/projected/335db604-5a37-42ad-a5b6-4536a8f5da96-kube-api-access-wbr4h\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.705608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705470 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-conf-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.705608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705494 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-system-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.705608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705527 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:20.705608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705590 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0906409-dea7-4779-9d86-4c9bfee0ee91-host\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.705608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705600 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f957aba3-d3ed-48a0-aff7-2c01130e95ee-serving-cert\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:21:20.705608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705616 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-cnibin\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705647 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705684 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-tmp\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705722 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705748 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-tmp\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705758 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705794 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-kubelet-config\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.705849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705826 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xc4q\" (UniqueName: \"kubernetes.io/projected/f957aba3-d3ed-48a0-aff7-2c01130e95ee-kube-api-access-8xc4q\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705855 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-ovn\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705880 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705906 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705932 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705957 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db87f267-b523-4a65-ab4c-b343a2d9ace5-config-volume\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.705981 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-service-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706016 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-config\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706066 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706091 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce43312-6761-45de-8350-d347681cb539-tmp-dir\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706118 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706140 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706140 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706160 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5010af-7674-4de3-ac32-4799caaf4f39-cert\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706166 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce43312-6761-45de-8350-d347681cb539-tmp-dir\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706211 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706279 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-trusted-ca\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.706339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706357 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-systemd-units\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706374 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4kdr\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706386 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706391 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-netns\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706419 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25de07e5-55c8-4c35-96af-7861b0a74469-data-volume\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706438 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-bin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706518 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25de07e5-55c8-4c35-96af-7861b0a74469-data-volume\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706481 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-agent-certs\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706565 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-device-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706590 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-etc-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706633 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706630 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12259580-c199-4cac-a5c2-35a79e28ca90-trusted-ca\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706650 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706665 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706688 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706761 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-bin\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706769 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706816 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-sys\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706860 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-multus\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706881 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-etc-kubernetes\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706904 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f67e5454-86f4-4e80-880a-070e4e7a38c1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.706944 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707018 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707066 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-lib-modules\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707104 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-konnectivity-ca\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707153 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707163 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707214 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f67e5454-86f4-4e80-880a-070e4e7a38c1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707249 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707282 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0906409-dea7-4779-9d86-4c9bfee0ee91-serviceca\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707300 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms9mk\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707314 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8jw\" (UniqueName: \"kubernetes.io/projected/f5824bdb-ee34-4c21-8655-5016b9338e5c-kube-api-access-9s8jw\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707341 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52aa29ef-34be-481f-b3a6-d78bfeb0b181-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707367 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-stats-auth\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707392 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707416 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707422 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707450 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89b5x\" (UniqueName: \"kubernetes.io/projected/12259580-c199-4cac-a5c2-35a79e28ca90-kube-api-access-89b5x\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707490 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclxl\" (UniqueName: \"kubernetes.io/projected/b6d33e27-7f70-4b3c-b90d-927c598a0630-kube-api-access-sclxl\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707516 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4nj\" (UniqueName: \"kubernetes.io/projected/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-kube-api-access-xh4nj\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707542 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysconfig\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707571 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52aa29ef-34be-481f-b3a6-d78bfeb0b181-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707565 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-kubernetes\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707620 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6kt\" (UniqueName: \"kubernetes.io/projected/db87f267-b523-4a65-ab4c-b343a2d9ace5-kube-api-access-hd6kt\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707636 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-stats-auth\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707649 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xfhz\"" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707658 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-snapshots\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.707598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707689 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zn6x\" (UniqueName: \"kubernetes.io/projected/8ce43312-6761-45de-8350-d347681cb539-kube-api-access-5zn6x\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707713 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tww9t\" (UniqueName: \"kubernetes.io/projected/25825c7b-a2c6-4263-bc62-d2c85c00ec47-kube-api-access-tww9t\") pod \"kas-connection-checker-685895c7c9-v4vbz\" (UID: \"25825c7b-a2c6-4263-bc62-d2c85c00ec47\") " pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707740 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/96367807-1ef3-4c30-b605-d041d6f53ae8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707764 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-dbus\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707793 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8vg\" (UniqueName: \"kubernetes.io/projected/f67e5454-86f4-4e80-880a-070e4e7a38c1-kube-api-access-wr8vg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707819 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-env-overrides\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707795 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707840 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707766 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6d33e27-7f70-4b3c-b90d-927c598a0630-snapshots\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707871 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707875 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xr757\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707876 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pk99b\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707796 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707868 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksmw\" (UniqueName: \"kubernetes.io/projected/d0906409-dea7-4779-9d86-4c9bfee0ee91-kube-api-access-cksmw\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707950 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gl4z9\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707955 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707957 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707975 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m5pkq\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.707979 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-tuned\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708041 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltprw\" (UniqueName: \"kubernetes.io/projected/25de07e5-55c8-4c35-96af-7861b0a74469-kube-api-access-ltprw\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708060 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-tuned\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708070 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx2g5\" (UniqueName: \"kubernetes.io/projected/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-kube-api-access-cx2g5\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708104 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-iptables-alerter-script\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708123 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-slash\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708148 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-systemd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708214 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-var-lib-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708249 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-conf\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708273 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-systemd\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708313 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p4b7h\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708310 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-multus-daemon-config\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708353 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-binary-copy\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708366 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708369 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-tmp\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708397 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-cnibin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708422 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/335db604-5a37-42ad-a5b6-4536a8f5da96-tmp\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708422 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00096998-7fc2-406c-b705-c4f80a552391-ovn-node-metrics-cert\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708448 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgj2q\" (UniqueName: \"kubernetes.io/projected/52aa29ef-34be-481f-b3a6-d78bfeb0b181-kube-api-access-dgj2q\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708484 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcp6k\" (UniqueName: \"kubernetes.io/projected/96818867-d92e-402c-b179-d93d211f45a1-kube-api-access-kcp6k\") pod \"network-check-source-6f78d59f99-76bkz\" (UID: \"96818867-d92e-402c-b179-d93d211f45a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708515 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-kubelet\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708545 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg5w\" (UniqueName: \"kubernetes.io/projected/474a81eb-340b-45fe-8033-0deaecf35f03-kube-api-access-csg5w\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708556 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w82k4\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708573 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708599 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-node-log\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708600 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00096998-7fc2-406c-b705-c4f80a552391-ovn-node-metrics-cert\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708623 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708729 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-var-lib-kubelet\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708752 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72cf5\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708763 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708795 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7t7\" (UniqueName: \"kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7\") pod \"migrator-66df5d496b-gpzw2\" (UID: \"c9621db0-ac02-43c2-9edd-ec7bd05d1817\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708807 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-szhfk\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708829 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5t2xq\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708847 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-27pxv\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708846 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708891 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f5824bdb-ee34-4c21-8655-5016b9338e5c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708930 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708948 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708953 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708980 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6wm76\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708998 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708982 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709021 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709033 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.708929 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709058 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709070 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709077 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709058 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709089 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709100 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709136 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12259580-c199-4cac-a5c2-35a79e28ca90-serving-cert\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709234 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709255 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709258 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709307 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709300 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709370 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709389 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12259580-c199-4cac-a5c2-35a79e28ca90-serving-cert\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709443 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709469 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709575 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709602 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709637 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709637 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709666 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.710508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.709638 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgxxf\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711370 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711426 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711437 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711464 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711509 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711511 5688 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711531 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711542 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711547 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711556 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711563 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711549 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711586 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711590 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711671 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:20.711727 5688 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711789 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.711843 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.712076 5688 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:20.712106 5688 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.us-west-1.compute.internal" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.712262 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 18:21:20.713598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.713406 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 18:21:20.716087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716066 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25de07e5-55c8-4c35-96af-7861b0a74469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716289 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-config\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.716481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716463 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5010af-7674-4de3-ac32-4799caaf4f39-cert\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:21:20.716631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716609 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/96367807-1ef3-4c30-b605-d041d6f53ae8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:20.716705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716613 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-metrics-certs\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.716705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716646 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db87f267-b523-4a65-ab4c-b343a2d9ace5-config-volume\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.716705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716610 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f17f5606-10aa-4f02-a2c4-16c28aba8540-default-certificate\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.716852 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716758 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96e03cbb-2909-4802-accd-002dd2922c42-original-pull-secret\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.716852 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716818 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-agent-certs\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:21:20.716852 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716828 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d33e27-7f70-4b3c-b90d-927c598a0630-serving-cert\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.717001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716837 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.717001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716830 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-cni-binary-copy\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.717001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716878 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.717001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.716983 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.717175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717006 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25de07e5-55c8-4c35-96af-7861b0a74469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.717175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717010 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-service-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.717175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717027 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.717175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717034 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f17f5606-10aa-4f02-a2c4-16c28aba8540-service-ca-bundle\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.717175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717011 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-ovnkube-script-lib\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.717175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717060 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-metrics-certs\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:20.717598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717580 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc74a8fe-8053-4b78-b8bf-569e5e34a89f-konnectivity-ca\") pod \"konnectivity-agent-v2bs5\" (UID: \"cc74a8fe-8053-4b78-b8bf-569e5e34a89f\") " pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:21:20.717659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717626 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.717659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717650 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db87f267-b523-4a65-ab4c-b343a2d9ace5-metrics-tls\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.717740 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717681 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.717902 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717884 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0906409-dea7-4779-9d86-4c9bfee0ee91-serviceca\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.718009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.717997 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00096998-7fc2-406c-b705-c4f80a552391-env-overrides\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.718124 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.718112 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/96367807-1ef3-4c30-b605-d041d6f53ae8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xncwq\" (UID: \"96367807-1ef3-4c30-b605-d041d6f53ae8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xncwq" Oct 31 18:21:20.718407 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.718391 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-iptables-alerter-script\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.718480 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.718468 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474a81eb-340b-45fe-8033-0deaecf35f03-multus-daemon-config\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.718603 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.718576 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f9210d-bf91-47c5-a4ab-318f8da1985d-cni-binary-copy\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.749475 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.749429 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 18:21:20.756037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.756022 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sqcvs\"" Oct 31 18:21:20.757308 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.757292 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d33e27-7f70-4b3c-b90d-927c598a0630-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.762027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.762009 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29596\" (UniqueName: \"kubernetes.io/projected/9d5010af-7674-4de3-ac32-4799caaf4f39-kube-api-access-29596\") pod \"ingress-canary-px79l\" (UID: \"9d5010af-7674-4de3-ac32-4799caaf4f39\") " pod="openshift-ingress-canary/ingress-canary-px79l" Oct 31 18:21:20.762099 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.762091 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpmx\" (UniqueName: \"kubernetes.io/projected/adfe1fd5-4835-4b2b-87d6-d218cfcd6b85-kube-api-access-4xpmx\") pod \"network-check-target-bch5w\" (UID: \"adfe1fd5-4835-4b2b-87d6-d218cfcd6b85\") " pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:21:20.763006 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.762984 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjtb\" (UniqueName: \"kubernetes.io/projected/4334dda9-cc0b-49bd-a5ca-48457df595fe-kube-api-access-6wjtb\") pod \"kas-connection-checker-685895c7c9-ksnkf\" (UID: \"4334dda9-cc0b-49bd-a5ca-48457df595fe\") " pod="kube-system/kas-connection-checker-685895c7c9-ksnkf" Oct 31 18:21:20.763177 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763159 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8dt\" (UniqueName: \"kubernetes.io/projected/f58038f2-47ce-43fc-bfa5-116eed7f4776-kube-api-access-lj8dt\") pod \"volume-data-source-validator-58c9f5f46d-98g5c\" (UID: \"f58038f2-47ce-43fc-bfa5-116eed7f4776\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-98g5c" Oct 31 18:21:20.763514 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763491 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xc4q\" (UniqueName: \"kubernetes.io/projected/f957aba3-d3ed-48a0-aff7-2c01130e95ee-kube-api-access-8xc4q\") pod \"service-ca-operator-544d749c8c-8pz8v\" (UID: \"f957aba3-d3ed-48a0-aff7-2c01130e95ee\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" Oct 31 18:21:20.763601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763516 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qp6\" (UniqueName: \"kubernetes.io/projected/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kube-api-access-n4qp6\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.763601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763528 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbr4h\" (UniqueName: \"kubernetes.io/projected/335db604-5a37-42ad-a5b6-4536a8f5da96-kube-api-access-wbr4h\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.763715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763646 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6xm\" (UniqueName: \"kubernetes.io/projected/00096998-7fc2-406c-b705-c4f80a552391-kube-api-access-bp6xm\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.763768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763745 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9qh\" (UniqueName: \"kubernetes.io/projected/f17f5606-10aa-4f02-a2c4-16c28aba8540-kube-api-access-9z9qh\") pod \"router-default-59bdcd5559-qvgdn\" (UID: \"f17f5606-10aa-4f02-a2c4-16c28aba8540\") " pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.763839 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763789 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.763977 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.763961 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd79\" (UniqueName: \"kubernetes.io/projected/89f9210d-bf91-47c5-a4ab-318f8da1985d-kube-api-access-nnd79\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.764262 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.764247 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrkf\" (UniqueName: \"kubernetes.io/projected/875a3d19-4e6c-4bb6-87b9-e801cd4f6604-kube-api-access-ggrkf\") pod \"kas-connection-checker-685895c7c9-xgpld\" (UID: \"875a3d19-4e6c-4bb6-87b9-e801cd4f6604\") " pod="kube-system/kas-connection-checker-685895c7c9-xgpld" Oct 31 18:21:20.766081 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.766067 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") pod \"image-registry-85cbfd9d9f-q2vwz\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:20.770840 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.770821 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8jw\" (UniqueName: \"kubernetes.io/projected/f5824bdb-ee34-4c21-8655-5016b9338e5c-kube-api-access-9s8jw\") pod \"cluster-samples-operator-6ddb6c8b94-z7w8t\" (UID: \"f5824bdb-ee34-4c21-8655-5016b9338e5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z7w8t" Oct 31 18:21:20.771394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.771377 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgj2q\" (UniqueName: \"kubernetes.io/projected/52aa29ef-34be-481f-b3a6-d78bfeb0b181-kube-api-access-dgj2q\") pod \"cluster-monitoring-operator-884576c57-wnxfz\" (UID: \"52aa29ef-34be-481f-b3a6-d78bfeb0b181\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wnxfz" Oct 31 18:21:20.771626 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.771608 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx2g5\" (UniqueName: \"kubernetes.io/projected/f22f6aaa-22cc-4ab9-a4bb-90d73eac861c-kube-api-access-cx2g5\") pod \"network-metrics-daemon-xkth6\" (UID: \"f22f6aaa-22cc-4ab9-a4bb-90d73eac861c\") " pod="openshift-multus/network-metrics-daemon-xkth6" Oct 31 18:21:20.771679 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.771654 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89b5x\" (UniqueName: \"kubernetes.io/projected/12259580-c199-4cac-a5c2-35a79e28ca90-kube-api-access-89b5x\") pod \"console-operator-7cd56b8dbc-zqcnd\" (UID: \"12259580-c199-4cac-a5c2-35a79e28ca90\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:20.771735 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.771687 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcp6k\" (UniqueName: \"kubernetes.io/projected/96818867-d92e-402c-b179-d93d211f45a1-kube-api-access-kcp6k\") pod \"network-check-source-6f78d59f99-76bkz\" (UID: \"96818867-d92e-402c-b179-d93d211f45a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-76bkz" Oct 31 18:21:20.771989 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.771974 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg5w\" (UniqueName: \"kubernetes.io/projected/474a81eb-340b-45fe-8033-0deaecf35f03-kube-api-access-csg5w\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.774279 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774261 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7t7\" (UniqueName: \"kubernetes.io/projected/c9621db0-ac02-43c2-9edd-ec7bd05d1817-kube-api-access-qj7t7\") pod \"migrator-66df5d496b-gpzw2\" (UID: \"c9621db0-ac02-43c2-9edd-ec7bd05d1817\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpzw2" Oct 31 18:21:20.774324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774262 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6kt\" (UniqueName: \"kubernetes.io/projected/db87f267-b523-4a65-ab4c-b343a2d9ace5-kube-api-access-hd6kt\") pod \"dns-default-pcvwd\" (UID: \"db87f267-b523-4a65-ab4c-b343a2d9ace5\") " pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:20.774652 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774626 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltprw\" (UniqueName: \"kubernetes.io/projected/25de07e5-55c8-4c35-96af-7861b0a74469-kube-api-access-ltprw\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.774874 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774853 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4nj\" (UniqueName: \"kubernetes.io/projected/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-kube-api-access-xh4nj\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.774952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774931 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tww9t\" (UniqueName: \"kubernetes.io/projected/25825c7b-a2c6-4263-bc62-d2c85c00ec47-kube-api-access-tww9t\") pod \"kas-connection-checker-685895c7c9-v4vbz\" (UID: \"25825c7b-a2c6-4263-bc62-d2c85c00ec47\") " pod="kube-system/kas-connection-checker-685895c7c9-v4vbz" Oct 31 18:21:20.775002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774964 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksmw\" (UniqueName: \"kubernetes.io/projected/d0906409-dea7-4779-9d86-4c9bfee0ee91-kube-api-access-cksmw\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.775002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774983 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8vg\" (UniqueName: \"kubernetes.io/projected/f67e5454-86f4-4e80-880a-070e4e7a38c1-kube-api-access-wr8vg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm\" (UID: \"f67e5454-86f4-4e80-880a-070e4e7a38c1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" Oct 31 18:21:20.775116 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.774965 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclxl\" (UniqueName: \"kubernetes.io/projected/b6d33e27-7f70-4b3c-b90d-927c598a0630-kube-api-access-sclxl\") pod \"insights-operator-7c4986d449-8z6s5\" (UID: \"b6d33e27-7f70-4b3c-b90d-927c598a0630\") " pod="openshift-insights/insights-operator-7c4986d449-8z6s5" Oct 31 18:21:20.776022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.776007 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zn6x\" (UniqueName: \"kubernetes.io/projected/8ce43312-6761-45de-8350-d347681cb539-kube-api-access-5zn6x\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.810008 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.809985 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-run\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810010 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-k8s-cni-cncf-io\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810026 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-netns\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810039 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-system-cni-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810054 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-modprobe-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810067 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810080 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-host\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810086 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-k8s-cni-cncf-io\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810101 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-multus-certs\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810113 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-run\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810123 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-system-cni-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810125 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-host-slash\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810157 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-multus-certs\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810156 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-run-netns\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810166 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-host\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810175 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-kubelet\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810130 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810222 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-hostroot\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810227 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759-host-slash\") pod \"iptables-alerter-rcrmh\" (UID: \"6ce9a5a8-8119-4fd2-baec-f2cb3b0d1759\") " pod="openshift-network-operator/iptables-alerter-rcrmh" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810232 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-modprobe-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810241 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-kubelet\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810250 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810257 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-hostroot\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810274 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-log-socket\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810295 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-netd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810315 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810332 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-log-socket\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810332 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25de07e5-55c8-4c35-96af-7861b0a74469-crio-socket\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810351 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-netd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810362 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-conf-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810374 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25de07e5-55c8-4c35-96af-7861b0a74469-crio-socket\") pod \"insights-runtime-extractor-np8hr\" (UID: \"25de07e5-55c8-4c35-96af-7861b0a74469\") " pod="openshift-insights/insights-runtime-extractor-np8hr" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810377 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-system-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810402 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-system-cni-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810420 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-conf-dir\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810412 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0906409-dea7-4779-9d86-4c9bfee0ee91-host\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810449 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-cnibin\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810477 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810494 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0906409-dea7-4779-9d86-4c9bfee0ee91-host\") pod \"node-ca-hf5q2\" (UID: \"d0906409-dea7-4779-9d86-4c9bfee0ee91\") " pod="openshift-image-registry/node-ca-hf5q2" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810501 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-kubelet-config\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810514 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810495 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-cnibin\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.810500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810522 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-ovn\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810531 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-kubelet-config\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810545 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810552 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-ovn\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810562 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810578 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-systemd-units\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810590 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-netns\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810604 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-bin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810616 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-device-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810641 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810643 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-netns\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810653 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-etc-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810669 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810678 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-bin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810686 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-bin\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810686 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-etc-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810704 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-systemd-units\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810666 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-device-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810716 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-cni-bin\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810742 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-sys\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810774 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-multus\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810800 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-sys\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810810 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-etc-kubernetes\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810817 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-host-var-lib-cni-multus\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810834 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810860 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-lib-modules\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810864 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-etc-kubernetes\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810887 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810893 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810911 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysconfig\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810932 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-kubernetes\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810956 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-dbus\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810977 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-lib-modules\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810980 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.810986 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-slash\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811014 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-kubernetes\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811013 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysconfig\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811033 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96e03cbb-2909-4802-accd-002dd2922c42-dbus\") pod \"global-pull-secret-syncer-tczvl\" (UID: \"96e03cbb-2909-4802-accd-002dd2922c42\") " pod="kube-system/global-pull-secret-syncer-tczvl" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811028 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-systemd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811054 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-slash\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811066 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-var-lib-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811072 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-run-systemd\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811089 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-conf\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811105 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-var-lib-openvswitch\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811111 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-systemd\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811158 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-cnibin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811170 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-systemd\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811177 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-kubelet\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811227 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-conf\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811208 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-host-kubelet\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811244 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-node-log\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811267 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00096998-7fc2-406c-b705-c4f80a552391-node-log\") pod \"ovnkube-node-pbjfs\" (UID: \"00096998-7fc2-406c-b705-c4f80a552391\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811268 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811236 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-cnibin\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811291 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-var-lib-kubelet\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811325 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811348 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811354 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-etc-sysctl-d\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811375 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ce43312-6761-45de-8350-d347681cb539-hosts-file\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811387 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f9210d-bf91-47c5-a4ab-318f8da1985d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g94x2\" (UID: \"89f9210d-bf91-47c5-a4ab-318f8da1985d\") " pod="openshift-multus/multus-additional-cni-plugins-g94x2" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811401 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811409 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811418 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ce43312-6761-45de-8350-d347681cb539-hosts-file\") pod \"node-resolver-z89h2\" (UID: \"8ce43312-6761-45de-8350-d347681cb539\") " pod="openshift-dns/node-resolver-z89h2" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811378 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/335db604-5a37-42ad-a5b6-4536a8f5da96-var-lib-kubelet\") pod \"tuned-sklpw\" (UID: \"335db604-5a37-42ad-a5b6-4536a8f5da96\") " pod="openshift-cluster-node-tuning-operator/tuned-sklpw" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811429 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-socket-dir-parent\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811457 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/512c0bfc-c41f-4abd-b4cd-c85d358fac3e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6f2s\" (UID: \"512c0bfc-c41f-4abd-b4cd-c85d358fac3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6f2s" Oct 31 18:21:20.811771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.811468 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474a81eb-340b-45fe-8033-0deaecf35f03-multus-socket-dir-parent\") pod \"multus-pz8h8\" (UID: \"474a81eb-340b-45fe-8033-0deaecf35f03\") " pod="openshift-multus/multus-pz8h8" Oct 31 18:21:20.904405 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.904379 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:20.914422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:20.914400 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 18:21:21.217804 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:21.217731 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcvwd" Oct 31 18:21:21.651541 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:21.651509 5688 generic.go:356] "Generic (PLEG): container finished" podID="f957aba3-d3ed-48a0-aff7-2c01130e95ee" containerID="35e74d9b3c83ca33dddf9a71efe99540d5d87bc12c840b201d3f378ea97e2157" exitCode=0 Oct 31 18:21:21.651995 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:21.651600 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" event={"ID":"f957aba3-d3ed-48a0-aff7-2c01130e95ee","Type":"ContainerDied","Data":"35e74d9b3c83ca33dddf9a71efe99540d5d87bc12c840b201d3f378ea97e2157"} Oct 31 18:21:21.652349 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:21.652334 5688 scope.go:122] "RemoveContainer" containerID="35e74d9b3c83ca33dddf9a71efe99540d5d87bc12c840b201d3f378ea97e2157" Oct 31 18:21:21.905799 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:21.905737 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:22.657981 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:22.657942 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8pz8v" event={"ID":"f957aba3-d3ed-48a0-aff7-2c01130e95ee","Type":"ContainerStarted","Data":"e628b0a23f943c25e6b6eddd8aacdc9740ad578a24928ddec421eca50d0b3221"} Oct 31 18:21:22.660019 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:22.659991 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np8hr" event={"ID":"25de07e5-55c8-4c35-96af-7861b0a74469","Type":"ContainerStarted","Data":"e00fc1574d0c30e64e2b2b5138952f609640031a3a8d3584c34adba325a0a502"} Oct 31 18:21:24.399515 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.399483 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:24.628012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.627979 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:24.628174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.628099 5688 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 18:21:24.629268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.629251 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59bdcd5559-qvgdn" Oct 31 18:21:24.669833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.669764 5688 generic.go:356] "Generic (PLEG): container finished" podID="f67e5454-86f4-4e80-880a-070e4e7a38c1" containerID="584f8d5e7733aca9d5f3041c2524ecd48dd285e990c4fa5dc7f0580a68e29a76" exitCode=0 Oct 31 18:21:24.669952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.669834 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" event={"ID":"f67e5454-86f4-4e80-880a-070e4e7a38c1","Type":"ContainerDied","Data":"584f8d5e7733aca9d5f3041c2524ecd48dd285e990c4fa5dc7f0580a68e29a76"} Oct 31 18:21:24.670157 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.670140 5688 scope.go:122] "RemoveContainer" containerID="584f8d5e7733aca9d5f3041c2524ecd48dd285e990c4fa5dc7f0580a68e29a76" Oct 31 18:21:24.671607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.671503 5688 generic.go:356] "Generic (PLEG): container finished" podID="b6d33e27-7f70-4b3c-b90d-927c598a0630" containerID="9d817cb6345d8060b037d37cc11b381dd002bd41574077b0cf7e14b98e94ed47" exitCode=0 Oct 31 18:21:24.672094 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.672071 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" event={"ID":"b6d33e27-7f70-4b3c-b90d-927c598a0630","Type":"ContainerDied","Data":"9d817cb6345d8060b037d37cc11b381dd002bd41574077b0cf7e14b98e94ed47"} Oct 31 18:21:24.672442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:24.672424 5688 scope.go:122] "RemoveContainer" containerID="9d817cb6345d8060b037d37cc11b381dd002bd41574077b0cf7e14b98e94ed47" Oct 31 18:21:25.215469 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:25.215439 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:25.677875 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:25.677840 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wqfmm" event={"ID":"f67e5454-86f4-4e80-880a-070e4e7a38c1","Type":"ContainerStarted","Data":"43e9aa5c53677c31053e3f75df9033769eb11e68a2eae1329a11e724421ef8f4"} Oct 31 18:21:25.679588 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:25.679565 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8z6s5" event={"ID":"b6d33e27-7f70-4b3c-b90d-927c598a0630","Type":"ContainerStarted","Data":"3a441b250695209324393f194f3f7118b3f2cc06012c81e2fa353b57b3038f52"} Oct 31 18:21:26.685386 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:26.685351 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np8hr" event={"ID":"25de07e5-55c8-4c35-96af-7861b0a74469","Type":"ContainerStarted","Data":"822d26b05ab089792ad25782157d461bb85e9a69d61c8199d4522a6c8f2209e1"} Oct 31 18:21:26.785170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:26.785146 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:26.785375 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:26.785363 5688 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 18:21:27.400464 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:27.400431 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v2bs5" Oct 31 18:21:27.973056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:27.972981 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r55xr"] Oct 31 18:21:27.975135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:27.975114 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.048919 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.048882 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 18:21:28.070370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070337 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-tls\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070364 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070384 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-wtmp\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070404 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-textfile\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070529 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-root\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070558 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08606964-6c14-4183-ba74-4973e60e692b-metrics-client-ca\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070584 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-sys\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070779 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070632 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-accelerators-collector-config\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.070779 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.070685 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jww\" (UniqueName: \"kubernetes.io/projected/08606964-6c14-4183-ba74-4973e60e692b-kube-api-access-l4jww\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.076670 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.076653 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 18:21:28.076670 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.076661 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9l6d9\"" Oct 31 18:21:28.126616 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.126571 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 18:21:28.126616 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.126591 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 18:21:28.171023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.170992 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-tls\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171025 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171045 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-wtmp\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171066 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-textfile\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171083 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-root\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171138 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-root\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171169 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08606964-6c14-4183-ba74-4973e60e692b-metrics-client-ca\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171249 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-sys\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171247 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-wtmp\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171274 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-accelerators-collector-config\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171317 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jww\" (UniqueName: \"kubernetes.io/projected/08606964-6c14-4183-ba74-4973e60e692b-kube-api-access-l4jww\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171324 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08606964-6c14-4183-ba74-4973e60e692b-sys\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171473 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-textfile\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.171746 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171635 5688 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 18:21:28.171968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.171936 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08606964-6c14-4183-ba74-4973e60e692b-metrics-client-ca\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.172165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.172147 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-accelerators-collector-config\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.173814 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.173795 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.173902 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.173883 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08606964-6c14-4183-ba74-4973e60e692b-node-exporter-tls\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.228953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.228891 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jww\" (UniqueName: \"kubernetes.io/projected/08606964-6c14-4183-ba74-4973e60e692b-kube-api-access-l4jww\") pod \"node-exporter-r55xr\" (UID: \"08606964-6c14-4183-ba74-4973e60e692b\") " pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.286971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.286940 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r55xr" Oct 31 18:21:28.298741 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:21:28.298718 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08606964_6c14_4183_ba74_4973e60e692b.slice/crio-c44b23ec9869749753fafca1c968e229395487aa88372759d6ad465e78ba04fc WatchSource:0}: Error finding container c44b23ec9869749753fafca1c968e229395487aa88372759d6ad465e78ba04fc: Status 404 returned error can't find the container with id c44b23ec9869749753fafca1c968e229395487aa88372759d6ad465e78ba04fc Oct 31 18:21:28.693137 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.693101 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r55xr" event={"ID":"08606964-6c14-4183-ba74-4973e60e692b","Type":"ContainerStarted","Data":"c44b23ec9869749753fafca1c968e229395487aa88372759d6ad465e78ba04fc"} Oct 31 18:21:28.696629 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:28.696602 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bch5w" Oct 31 18:21:29.565718 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:29.565693 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:29.566174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:29.565749 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zqcnd" Oct 31 18:21:30.701968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:30.701927 5688 generic.go:356] "Generic (PLEG): container finished" podID="08606964-6c14-4183-ba74-4973e60e692b" containerID="a835789e17016a0324361ba9375fc31ba4eb07d0687ff8201ac0d8b381cd18ea" exitCode=0 Oct 31 18:21:30.702396 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:30.701983 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r55xr" event={"ID":"08606964-6c14-4183-ba74-4973e60e692b","Type":"ContainerDied","Data":"a835789e17016a0324361ba9375fc31ba4eb07d0687ff8201ac0d8b381cd18ea"} Oct 31 18:21:31.707365 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:31.707330 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r55xr" event={"ID":"08606964-6c14-4183-ba74-4973e60e692b","Type":"ContainerStarted","Data":"7fb1079c072359c7f9531596466d532628961c31d5fcf87df43c3a6f64cffbd5"} Oct 31 18:21:31.707752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:31.707382 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r55xr" event={"ID":"08606964-6c14-4183-ba74-4973e60e692b","Type":"ContainerStarted","Data":"ea7949689e21c4fb66a1a13ddb042f2dd4b4321dd95b667a1a74bcf6a31e67a2"} Oct 31 18:21:31.877639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:31.877585 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r55xr" podStartSLOduration=3.241076493 podStartE2EDuration="4.877571592s" totalImagesPullingTime="1.559974143s" totalInitContainerRuntime="76.520956ms" isStatefulPod=true podCreationTimestamp="2026-10-31 18:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:31.876172802 +0000 UTC m=+12.478313938" watchObservedRunningTime="2026-10-31 18:21:31.877571592 +0000 UTC m=+12.479712735" Oct 31 18:21:33.308983 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:33.308952 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85cbfd9d9f-q2vwz"] Oct 31 18:21:33.313556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:33.313533 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:38.451796 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:38.451754 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbjfs" Oct 31 18:21:54.906536 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:54.906500 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:21:58.329622 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.329542 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" podUID="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" containerName="registry" containerID="cri-o://6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7" gracePeriod=30 Oct 31 18:21:58.585424 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.585345 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:58.669988 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.669960 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670007 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670031 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670053 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670227 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670299 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670349 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670402 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") pod \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\" (UID: \"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5\") " Oct 31 18:21:58.670640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670593 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:21:58.670720 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.670675 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:21:58.672747 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.672718 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:21:58.673270 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.673246 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:21:58.673370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.673305 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:21:58.673370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.673352 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "kube-api-access-k2vvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:21:58.673486 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.673420 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:21:58.676791 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.676766 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted" pod "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" (UID: "08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:21:58.771483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771450 5688 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-installation-pull-secrets\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771476 5688 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-bound-sa-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771485 5688 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-trusted-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771494 5688 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-image-registry-private-configuration\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771504 5688 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-ca-trust-extracted\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771512 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vvm\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-kube-api-access-k2vvm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771520 5688 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-certificates\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.771734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.771528 5688 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5-registry-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:21:58.807206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.807148 5688 generic.go:356] "Generic (PLEG): container finished" podID="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" containerID="6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7" exitCode=0 Oct 31 18:21:58.807388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.807227 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" Oct 31 18:21:58.807388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.807227 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" event={"ID":"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5","Type":"ContainerDied","Data":"6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7"} Oct 31 18:21:58.807388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.807301 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cbfd9d9f-q2vwz" event={"ID":"08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5","Type":"ContainerDied","Data":"2da5603647dd25425daee5a88755a7a281087a098101bbb80ffcf1307d6dc717"} Oct 31 18:21:58.807388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.807323 5688 scope.go:122] "RemoveContainer" containerID="6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7" Oct 31 18:21:58.816597 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.816582 5688 scope.go:122] "RemoveContainer" containerID="6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7" Oct 31 18:21:58.816871 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:21:58.816849 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7\": container with ID starting with 6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7 not found: ID does not exist" containerID="6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7" Oct 31 18:21:58.816924 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.816882 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7"} err="failed to get container status \"6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7\": rpc error: code = NotFound desc = could not find container \"6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7\": container with ID starting with 6b14f9487eb7ba82d9ffb50933d7e82a94f9cc7e741e4809258342fa0005b7d7 not found: ID does not exist" Oct 31 18:21:58.970749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.970674 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85cbfd9d9f-q2vwz"] Oct 31 18:21:58.974496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:58.974472 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85cbfd9d9f-q2vwz"] Oct 31 18:21:59.564819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:21:59.564788 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" path="/var/lib/kubelet/pods/08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5/volumes" Oct 31 18:22:24.905598 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:22:24.905558 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:22:54.903765 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:22:54.903721 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:23:24.904656 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:23:24.904571 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:23:54.903526 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:23:54.903488 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:24:24.903849 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:24:24.903810 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:24:37.730934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.730885 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2"] Oct 31 18:24:37.731394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.731336 5688 cpu_manager.go:414] "containerMap: removing container" podUID="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" containerName="registry" Oct 31 18:24:37.731394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.731357 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" containerName="registry" Oct 31 18:24:37.731514 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.731455 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c5f7ab-a620-4ea1-b358-cf6e35bc6ba5" containerName="registry" Oct 31 18:24:37.734411 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.734394 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778060 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778030 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778067 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778159 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778220 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778252 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778431 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778279 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.778431 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.778372 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxgk\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.789353 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.789326 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 18:24:37.838473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.838439 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 18:24:37.838650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.838445 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 18:24:37.838650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.838438 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 18:24:37.838650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.838517 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vkrfg\"" Oct 31 18:24:37.839220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.839202 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2"] Oct 31 18:24:37.878872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.878833 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.878890 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.878929 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.878945 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.878962 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.878988 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.879024 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxgk\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.879786 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.879685 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.881488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.881466 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.881488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.881480 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.881650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.881628 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.881818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.881796 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.936043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.936006 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:37.936200 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:37.936006 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxgk\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk\") pod \"istiod-openshift-gateway-7876bfcb74-xrvv2\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:38.045594 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:38.045558 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:38.471931 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:38.471861 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2"] Oct 31 18:24:38.475150 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:24:38.475123 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728c1b36_a297_4ff2_9a6b_c49816580ed6.slice/crio-286b6934caa8fa8dcd3d595f6c0371f53b3734f83c217e063054bdf4557ed37b WatchSource:0}: Error finding container 286b6934caa8fa8dcd3d595f6c0371f53b3734f83c217e063054bdf4557ed37b: Status 404 returned error can't find the container with id 286b6934caa8fa8dcd3d595f6c0371f53b3734f83c217e063054bdf4557ed37b Oct 31 18:24:39.374430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:39.374390 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" event={"ID":"728c1b36-a297-4ff2-9a6b-c49816580ed6","Type":"ContainerStarted","Data":"286b6934caa8fa8dcd3d595f6c0371f53b3734f83c217e063054bdf4557ed37b"} Oct 31 18:24:42.672272 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:42.672232 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:24:42.672512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:42.672307 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:24:43.392883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:43.392847 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" event={"ID":"728c1b36-a297-4ff2-9a6b-c49816580ed6","Type":"ContainerStarted","Data":"fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074"} Oct 31 18:24:43.393092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:43.393061 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:43.394825 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:43.394795 5688 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xrvv2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:24:43.394940 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:43.394855 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:24:43.599784 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:43.599730 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" podStartSLOduration=2.4051875750000002 podStartE2EDuration="6.599715399s" totalImagesPullingTime="4.194527824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:24:43.559155501 +0000 UTC m=+204.161296678" watchObservedRunningTime="2026-10-31 18:24:43.599715399 +0000 UTC m=+204.201856544" Oct 31 18:24:44.396923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:44.396882 5688 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xrvv2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:24:44.397424 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:44.396953 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:24:47.397235 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:24:47.397200 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:24:54.904255 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:24:54.904210 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:25:23.267956 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.267926 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7"] Oct 31 18:25:23.273793 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.273777 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.315007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.314967 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.315278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.315019 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.315278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.315112 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnw98\" (UniqueName: \"kubernetes.io/projected/e7f7a157-477a-4c30-a7b1-22887b2fe332-kube-api-access-mnw98\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.315278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.315155 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.315278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.315205 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f7a157-477a-4c30-a7b1-22887b2fe332-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.315649 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.315516 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7f7a157-477a-4c30-a7b1-22887b2fe332-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.318239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.318215 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7"] Oct 31 18:25:23.329413 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.329389 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-k5llp\"" Oct 31 18:25:23.416452 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416425 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.416572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416458 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.416572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416474 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnw98\" (UniqueName: \"kubernetes.io/projected/e7f7a157-477a-4c30-a7b1-22887b2fe332-kube-api-access-mnw98\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.416572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416500 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.416572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416523 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f7a157-477a-4c30-a7b1-22887b2fe332-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.416572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416564 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7f7a157-477a-4c30-a7b1-22887b2fe332-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.416894 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.416876 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7f7a157-477a-4c30-a7b1-22887b2fe332-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.418938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.418912 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f7a157-477a-4c30-a7b1-22887b2fe332-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.422845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.422813 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 18:25:23.422968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.422851 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 18:25:23.422968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.422880 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 18:25:23.422968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.422950 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 18:25:23.423142 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.423045 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 18:25:23.429473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.429450 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.429586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.429502 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.429586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.429502 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7f7a157-477a-4c30-a7b1-22887b2fe332-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.473466 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.473436 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnw98\" (UniqueName: \"kubernetes.io/projected/e7f7a157-477a-4c30-a7b1-22887b2fe332-kube-api-access-mnw98\") pod \"opendatahub-operator-controller-manager-dd8c87985-xhlp7\" (UID: \"e7f7a157-477a-4c30-a7b1-22887b2fe332\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:23.587864 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:23.587787 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:24.075786 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:24.075755 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7"] Oct 31 18:25:24.079161 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:25:24.079133 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f7a157_477a_4c30_a7b1_22887b2fe332.slice/crio-9ebef9b5420d764ad53bae82d1ececd21c5781acc6ea5ba1e78b0664814e5047 WatchSource:0}: Error finding container 9ebef9b5420d764ad53bae82d1ececd21c5781acc6ea5ba1e78b0664814e5047: Status 404 returned error can't find the container with id 9ebef9b5420d764ad53bae82d1ececd21c5781acc6ea5ba1e78b0664814e5047 Oct 31 18:25:24.535201 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:24.535143 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" event={"ID":"e7f7a157-477a-4c30-a7b1-22887b2fe332","Type":"ContainerStarted","Data":"9ebef9b5420d764ad53bae82d1ececd21c5781acc6ea5ba1e78b0664814e5047"} Oct 31 18:25:24.903716 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:25:24.903629 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:25:28.556530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:28.556439 5688 generic.go:356] "Generic (PLEG): container finished" podID="e7f7a157-477a-4c30-a7b1-22887b2fe332" containerID="17e303770a36f69e10bb6b1d2862a755ea9c9495a45de4e32fee9f7f26bf9fbe" exitCode=0 Oct 31 18:25:28.556530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:28.556508 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" event={"ID":"e7f7a157-477a-4c30-a7b1-22887b2fe332","Type":"ContainerDied","Data":"17e303770a36f69e10bb6b1d2862a755ea9c9495a45de4e32fee9f7f26bf9fbe"} Oct 31 18:25:29.564376 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:29.564347 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" event={"ID":"e7f7a157-477a-4c30-a7b1-22887b2fe332","Type":"ContainerStarted","Data":"9b736990d212afe5b0669d8c23dd974b020f69176c87f9a818683494def5067b"} Oct 31 18:25:29.564786 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:29.564471 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:29.797875 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:29.797808 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" podStartSLOduration=1.880105719 podStartE2EDuration="6.797790163s" totalImagesPullingTime="4.505920077s" totalInitContainerRuntime="411.764367ms" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:29.747794581 +0000 UTC m=+250.349935727" watchObservedRunningTime="2026-10-31 18:25:29.797790163 +0000 UTC m=+250.399931308" Oct 31 18:25:40.570318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:40.570290 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-xhlp7" Oct 31 18:25:54.903931 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:25:54.903895 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:25:59.670296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.670224 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g"] Oct 31 18:25:59.673857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.673833 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.708416 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.708386 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a491c77-ee30-48ea-b10f-8f700ff7d850-tmp\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.708545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.708442 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a491c77-ee30-48ea-b10f-8f700ff7d850-tls-certs\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.708545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.708530 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmd8h\" (UniqueName: \"kubernetes.io/projected/8a491c77-ee30-48ea-b10f-8f700ff7d850-kube-api-access-fmd8h\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.725756 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.725730 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 18:25:59.725756 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.725747 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 18:25:59.775836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.775811 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hzmzn\"" Oct 31 18:25:59.776690 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.776663 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g"] Oct 31 18:25:59.808859 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.808832 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a491c77-ee30-48ea-b10f-8f700ff7d850-tmp\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.808972 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.808872 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a491c77-ee30-48ea-b10f-8f700ff7d850-tls-certs\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.808972 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.808907 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmd8h\" (UniqueName: \"kubernetes.io/projected/8a491c77-ee30-48ea-b10f-8f700ff7d850-kube-api-access-fmd8h\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.811231 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.811209 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a491c77-ee30-48ea-b10f-8f700ff7d850-tmp\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.811549 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.811531 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a491c77-ee30-48ea-b10f-8f700ff7d850-tls-certs\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.866773 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.866749 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmd8h\" (UniqueName: \"kubernetes.io/projected/8a491c77-ee30-48ea-b10f-8f700ff7d850-kube-api-access-fmd8h\") pod \"kube-auth-proxy-5d4f97d94f-8g88g\" (UID: \"8a491c77-ee30-48ea-b10f-8f700ff7d850\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:25:59.985751 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:25:59.985724 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" Oct 31 18:26:00.360841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:00.360762 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g"] Oct 31 18:26:00.364575 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:26:00.364544 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a491c77_ee30_48ea_b10f_8f700ff7d850.slice/crio-804038a7c6a9f0abddfd9c084bbdbe33ca3f966b45526f41f79c8ea6f6e70997 WatchSource:0}: Error finding container 804038a7c6a9f0abddfd9c084bbdbe33ca3f966b45526f41f79c8ea6f6e70997: Status 404 returned error can't find the container with id 804038a7c6a9f0abddfd9c084bbdbe33ca3f966b45526f41f79c8ea6f6e70997 Oct 31 18:26:00.678672 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:00.678594 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" event={"ID":"8a491c77-ee30-48ea-b10f-8f700ff7d850","Type":"ContainerStarted","Data":"804038a7c6a9f0abddfd9c084bbdbe33ca3f966b45526f41f79c8ea6f6e70997"} Oct 31 18:26:05.709703 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:05.709670 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" event={"ID":"8a491c77-ee30-48ea-b10f-8f700ff7d850","Type":"ContainerStarted","Data":"4b1f1411f5544177164972a6de1039072ad82ba7b9f80a7cca363e12c1d630a6"} Oct 31 18:26:05.825256 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:05.825202 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8g88g" podStartSLOduration=2.535695029 podStartE2EDuration="6.825173966s" totalImagesPullingTime="4.289478937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:05.824487313 +0000 UTC m=+286.426628457" watchObservedRunningTime="2026-10-31 18:26:05.825173966 +0000 UTC m=+286.427315108" Oct 31 18:26:12.011251 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.011219 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf"] Oct 31 18:26:12.015594 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.015578 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.103063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.103031 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.103235 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.103071 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rzv\" (UniqueName: \"kubernetes.io/projected/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-kube-api-access-f2rzv\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.103235 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.103197 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.119529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.119498 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 18:26:12.119529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.119506 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 18:26:12.119724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.119512 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cfcnz\"" Oct 31 18:26:12.119724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.119514 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 18:26:12.119724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.119545 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 18:26:12.120544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.120524 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf"] Oct 31 18:26:12.204417 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.204388 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.204551 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.204430 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.204551 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.204465 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rzv\" (UniqueName: \"kubernetes.io/projected/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-kube-api-access-f2rzv\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.205219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.205127 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.206930 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.206889 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.260558 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.260533 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rzv\" (UniqueName: \"kubernetes.io/projected/5c4762c4-658b-4a35-bfc5-b89a29bb2e11-kube-api-access-f2rzv\") pod \"kuadrant-console-plugin-66c4fb575d-4ddzf\" (UID: \"5c4762c4-658b-4a35-bfc5-b89a29bb2e11\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.339833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.339764 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" Oct 31 18:26:12.706563 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.706492 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf"] Oct 31 18:26:12.710072 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:26:12.710046 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c4762c4_658b_4a35_bfc5_b89a29bb2e11.slice/crio-a66e1f477a297b7a2388343cadd7bccd7abcabc857ef9d121fab54c02cb4598d WatchSource:0}: Error finding container a66e1f477a297b7a2388343cadd7bccd7abcabc857ef9d121fab54c02cb4598d: Status 404 returned error can't find the container with id a66e1f477a297b7a2388343cadd7bccd7abcabc857ef9d121fab54c02cb4598d Oct 31 18:26:12.737295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:12.737266 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" event={"ID":"5c4762c4-658b-4a35-bfc5-b89a29bb2e11","Type":"ContainerStarted","Data":"a66e1f477a297b7a2388343cadd7bccd7abcabc857ef9d121fab54c02cb4598d"} Oct 31 18:26:19.580399 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:19.580372 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:26:19.581959 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:19.581934 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:26:24.907264 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:26:24.907219 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:26:25.795356 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:25.795301 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" event={"ID":"5c4762c4-658b-4a35-bfc5-b89a29bb2e11","Type":"ContainerStarted","Data":"5eeb7486455a1f4d4a8241bf88e98a6b1172cec26995e77988eb6f0fc1032296"} Oct 31 18:26:25.908791 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:25.908718 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4ddzf" podStartSLOduration=2.274878048 podStartE2EDuration="14.908699926s" totalImagesPullingTime="12.633821878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:25.907519466 +0000 UTC m=+306.509660611" watchObservedRunningTime="2026-10-31 18:26:25.908699926 +0000 UTC m=+306.510841074" Oct 31 18:26:52.200914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.200878 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c7fc58f4c-msxcq"] Oct 31 18:26:52.207288 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.207266 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.260087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.260056 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 18:26:52.309740 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.309707 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 18:26:52.309920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.309892 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 18:26:52.310037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.310019 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 18:26:52.310037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.310027 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 18:26:52.310146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.310062 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-clhwj\"" Oct 31 18:26:52.311117 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.311101 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c7fc58f4c-msxcq"] Oct 31 18:26:52.355532 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.355498 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.355680 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.355540 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.355680 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.355580 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9rn\" (UniqueName: \"kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.456527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.456442 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.456527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.456488 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.456527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.456516 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9rn\" (UniqueName: \"kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.459565 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.459542 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.459677 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.459604 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.513413 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.513378 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9rn\" (UniqueName: \"kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn\") pod \"maas-controller-7c7fc58f4c-msxcq\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.520210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.520169 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:26:52.780872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.780839 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-hx8bn"] Oct 31 18:26:52.784348 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.784327 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:52.842978 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.842938 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-hx8bn"] Oct 31 18:26:52.905937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.905905 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c7fc58f4c-msxcq"] Oct 31 18:26:52.910274 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:26:52.910241 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a5b9d7_5d84_472c_8210_8eb3f31262dc.slice/crio-5be7fdaf4fc9f86155d1bebb21f0761349a3b6a0053bdb741e40ba892765816e WatchSource:0}: Error finding container 5be7fdaf4fc9f86155d1bebb21f0761349a3b6a0053bdb741e40ba892765816e: Status 404 returned error can't find the container with id 5be7fdaf4fc9f86155d1bebb21f0761349a3b6a0053bdb741e40ba892765816e Oct 31 18:26:52.911511 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.911491 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 18:26:52.961262 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.961230 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:52.961410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.961266 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnl7\" (UniqueName: \"kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:52.961410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:52.961399 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.062882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.062784 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.062882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.062828 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.062882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.062846 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnl7\" (UniqueName: \"kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.065455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.065429 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.065554 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.065456 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.120050 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.120018 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnl7\" (UniqueName: \"kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7\") pod \"maas-controller-64fbfd7b7f-hx8bn\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.397240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.397143 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:53.765530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.765486 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-hx8bn"] Oct 31 18:26:53.768497 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:26:53.768473 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43418aaa_e973_435b_b950_f23e7d30e5a8.slice/crio-9e68c68b9bf7304fce48e717aea6a83fc059c9d74091ebb17b851fd1726b7e3d WatchSource:0}: Error finding container 9e68c68b9bf7304fce48e717aea6a83fc059c9d74091ebb17b851fd1726b7e3d: Status 404 returned error can't find the container with id 9e68c68b9bf7304fce48e717aea6a83fc059c9d74091ebb17b851fd1726b7e3d Oct 31 18:26:53.904512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.904476 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" event={"ID":"65a5b9d7-5d84-472c-8210-8eb3f31262dc","Type":"ContainerStarted","Data":"5be7fdaf4fc9f86155d1bebb21f0761349a3b6a0053bdb741e40ba892765816e"} Oct 31 18:26:53.905607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:53.905569 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" event={"ID":"43418aaa-e973-435b-b950-f23e7d30e5a8","Type":"ContainerStarted","Data":"9e68c68b9bf7304fce48e717aea6a83fc059c9d74091ebb17b851fd1726b7e3d"} Oct 31 18:26:54.912123 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:26:54.912078 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:26:59.938503 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:59.938463 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" event={"ID":"43418aaa-e973-435b-b950-f23e7d30e5a8","Type":"ContainerStarted","Data":"740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea"} Oct 31 18:26:59.938965 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:59.938572 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:26:59.939999 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:59.939964 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" event={"ID":"65a5b9d7-5d84-472c-8210-8eb3f31262dc","Type":"ContainerStarted","Data":"462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8"} Oct 31 18:26:59.940116 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:26:59.940083 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:27:00.061545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:00.061491 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" podStartSLOduration=2.419904788 podStartE2EDuration="8.061476423s" totalImagesPullingTime="5.641571635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:00.06054605 +0000 UTC m=+340.662687226" watchObservedRunningTime="2026-10-31 18:27:00.061476423 +0000 UTC m=+340.663617566" Oct 31 18:27:00.177500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:00.177445 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" podStartSLOduration=1.685385406 podStartE2EDuration="8.177429753s" totalImagesPullingTime="6.492044347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:00.17658906 +0000 UTC m=+340.778730205" watchObservedRunningTime="2026-10-31 18:27:00.177429753 +0000 UTC m=+340.779570939" Oct 31 18:27:10.946334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:10.946304 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 18:27:10.946788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:10.946376 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:27:11.131200 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.131152 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c7fc58f4c-msxcq"] Oct 31 18:27:11.132403 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.131376 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" podUID="65a5b9d7-5d84-472c-8210-8eb3f31262dc" containerName="manager" containerID="cri-o://462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8" gracePeriod=30 Oct 31 18:27:11.369822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.369799 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:27:11.419669 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.419641 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert\") pod \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " Oct 31 18:27:11.419799 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.419699 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs\") pod \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " Oct 31 18:27:11.419799 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.419722 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn\" (UniqueName: \"kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn\") pod \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\" (UID: \"65a5b9d7-5d84-472c-8210-8eb3f31262dc\") " Oct 31 18:27:11.421941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.421914 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert" pod "65a5b9d7-5d84-472c-8210-8eb3f31262dc" (UID: "65a5b9d7-5d84-472c-8210-8eb3f31262dc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:11.422088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.422070 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn" pod "65a5b9d7-5d84-472c-8210-8eb3f31262dc" (UID: "65a5b9d7-5d84-472c-8210-8eb3f31262dc"). InnerVolumeSpecName "kube-api-access-fz9rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:27:11.422088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.422072 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs" pod "65a5b9d7-5d84-472c-8210-8eb3f31262dc" (UID: "65a5b9d7-5d84-472c-8210-8eb3f31262dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:11.520979 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.520893 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:11.520979 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.520922 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9rn\" (UniqueName: \"kubernetes.io/projected/65a5b9d7-5d84-472c-8210-8eb3f31262dc-kube-api-access-fz9rn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:11.520979 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.520933 5688 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a5b9d7-5d84-472c-8210-8eb3f31262dc-webhook-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:11.988664 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.988628 5688 generic.go:356] "Generic (PLEG): container finished" podID="65a5b9d7-5d84-472c-8210-8eb3f31262dc" containerID="462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8" exitCode=0 Oct 31 18:27:11.989092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.988672 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" Oct 31 18:27:11.989092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.988674 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" event={"ID":"65a5b9d7-5d84-472c-8210-8eb3f31262dc","Type":"ContainerDied","Data":"462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8"} Oct 31 18:27:11.989092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.988806 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c7fc58f4c-msxcq" event={"ID":"65a5b9d7-5d84-472c-8210-8eb3f31262dc","Type":"ContainerDied","Data":"5be7fdaf4fc9f86155d1bebb21f0761349a3b6a0053bdb741e40ba892765816e"} Oct 31 18:27:11.989092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.988830 5688 scope.go:122] "RemoveContainer" containerID="462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8" Oct 31 18:27:11.997008 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.996991 5688 scope.go:122] "RemoveContainer" containerID="462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8" Oct 31 18:27:11.997306 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:27:11.997278 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8\": container with ID starting with 462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8 not found: ID does not exist" containerID="462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8" Oct 31 18:27:11.997372 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:11.997306 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8"} err="failed to get container status \"462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8\": rpc error: code = NotFound desc = could not find container \"462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8\": container with ID starting with 462e1af6de8eb75b5d77c7f43e3bfd61b846edb4646391fd37897cd4c551b9d8 not found: ID does not exist" Oct 31 18:27:12.158216 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:12.158166 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c7fc58f4c-msxcq"] Oct 31 18:27:12.168095 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:12.168068 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c7fc58f4c-msxcq"] Oct 31 18:27:13.563619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.563583 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a5b9d7-5d84-472c-8210-8eb3f31262dc" path="/var/lib/kubelet/pods/65a5b9d7-5d84-472c-8210-8eb3f31262dc/volumes" Oct 31 18:27:13.764771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.764731 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv"] Oct 31 18:27:13.765121 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.765104 5688 cpu_manager.go:414] "containerMap: removing container" podUID="65a5b9d7-5d84-472c-8210-8eb3f31262dc" containerName="manager" Oct 31 18:27:13.765238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.765123 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a5b9d7-5d84-472c-8210-8eb3f31262dc" containerName="manager" Oct 31 18:27:13.765291 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.765274 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a5b9d7-5d84-472c-8210-8eb3f31262dc" containerName="manager" Oct 31 18:27:13.769565 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.769543 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841077 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.840997 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841077 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.841027 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltxn\" (UniqueName: \"kubernetes.io/projected/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-kube-api-access-5ltxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841077 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.841051 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.841105 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.841175 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.841251 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.841383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.841355 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.873119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.873086 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv"] Oct 31 18:27:13.942498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942462 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.942498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942501 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltxn\" (UniqueName: \"kubernetes.io/projected/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-kube-api-access-5ltxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.942757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942531 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.942757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942555 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.942757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942579 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.942757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942630 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.942757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.942673 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.943859 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.943828 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.945013 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.944980 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.945138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.945117 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.945233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.945211 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.945554 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.945537 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.999663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.999626 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:13.999938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:13.999913 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltxn\" (UniqueName: \"kubernetes.io/projected/16e4ddc8-e42c-46d6-8d9b-8784b06d4760-kube-api-access-5ltxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mlhcv\" (UID: \"16e4ddc8-e42c-46d6-8d9b-8784b06d4760\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:14.081537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:14.081500 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:14.506445 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:14.506419 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv"] Oct 31 18:27:14.509087 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:27:14.509047 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e4ddc8_e42c_46d6_8d9b_8784b06d4760.slice/crio-c163a722d35b9a42675c8ed45cba39fcb0282da2ee924e28e4e80f9d77cf95e9 WatchSource:0}: Error finding container c163a722d35b9a42675c8ed45cba39fcb0282da2ee924e28e4e80f9d77cf95e9: Status 404 returned error can't find the container with id c163a722d35b9a42675c8ed45cba39fcb0282da2ee924e28e4e80f9d77cf95e9 Oct 31 18:27:14.511159 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:14.511125 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:27:14.511237 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:14.511206 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:27:15.004709 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:15.004676 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" event={"ID":"16e4ddc8-e42c-46d6-8d9b-8784b06d4760","Type":"ContainerStarted","Data":"60cec640deca2f10af6b642fe8f6d8e920ee0c5418931b6ab27177695734a66d"} Oct 31 18:27:15.005088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:15.004728 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" event={"ID":"16e4ddc8-e42c-46d6-8d9b-8784b06d4760","Type":"ContainerStarted","Data":"c163a722d35b9a42675c8ed45cba39fcb0282da2ee924e28e4e80f9d77cf95e9"} Oct 31 18:27:15.005088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:15.004857 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:15.006505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:15.006481 5688 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mlhcv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:27:15.006605 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:15.006538 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" podUID="16e4ddc8-e42c-46d6-8d9b-8784b06d4760" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:27:15.156949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:15.156900 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" podStartSLOduration=2.156885973 podStartE2EDuration="2.156885973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:15.123011989 +0000 UTC m=+355.725153133" watchObservedRunningTime="2026-10-31 18:27:15.156885973 +0000 UTC m=+355.759027116" Oct 31 18:27:16.009980 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:16.009953 5688 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mlhcv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:27:16.010380 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:16.010002 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" podUID="16e4ddc8-e42c-46d6-8d9b-8784b06d4760" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:27:19.010806 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.010778 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mlhcv" Oct 31 18:27:19.228425 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.228392 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2"] Oct 31 18:27:19.228646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.228624 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerName="discovery" containerID="cri-o://fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074" gracePeriod=30 Oct 31 18:27:19.487450 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.487418 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:27:19.593846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593787 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.593846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593827 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.594026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593871 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.594026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593907 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.594026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593930 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.594026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593962 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.594026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.593989 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig\") pod \"728c1b36-a297-4ff2-9a6b-c49816580ed6\" (UID: \"728c1b36-a297-4ff2-9a6b-c49816580ed6\") " Oct 31 18:27:19.594304 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.594277 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:27:19.596217 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.596166 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:19.596336 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.596290 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:27:19.596596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.596566 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:27:19.596596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.596577 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "kube-api-access-zqxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:27:19.596725 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.596705 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:19.596762 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.596718 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig" pod "728c1b36-a297-4ff2-9a6b-c49816580ed6" (UID: "728c1b36-a297-4ff2-9a6b-c49816580ed6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:19.694722 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694690 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.694722 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694718 5688 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-ca-configmap\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.694926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694731 5688 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-cacerts\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.694926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694744 5688 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-csr-dns-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.694926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694754 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxgk\" (UniqueName: \"kubernetes.io/projected/728c1b36-a297-4ff2-9a6b-c49816580ed6-kube-api-access-zqxgk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.694926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694765 5688 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/728c1b36-a297-4ff2-9a6b-c49816580ed6-istio-kubeconfig\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.694926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.694776 5688 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c1b36-a297-4ff2-9a6b-c49816580ed6-local-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:19.738626 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.738592 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5"] Oct 31 18:27:19.738966 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.738949 5688 cpu_manager.go:414] "containerMap: removing container" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerName="discovery" Oct 31 18:27:19.738966 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.738965 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerName="discovery" Oct 31 18:27:19.739137 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.739021 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerName="discovery" Oct 31 18:27:19.743332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.743312 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.795482 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.795445 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8\") pod \"llmisvc-controller-manager-869d5dcc5d-2qfz5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.795655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.795508 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2qfz5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.800940 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.800916 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5"] Oct 31 18:27:19.810450 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.810427 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 18:27:19.810626 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.810609 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mzlbg\"" Oct 31 18:27:19.896242 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.896147 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8\") pod \"llmisvc-controller-manager-869d5dcc5d-2qfz5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.896242 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.896236 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2qfz5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.898730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.898713 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2qfz5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.953682 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.953594 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8\") pod \"llmisvc-controller-manager-869d5dcc5d-2qfz5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:19.954286 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.954262 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-cgxbd"] Oct 31 18:27:19.957496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:19.957480 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.010333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.010310 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 18:27:20.010333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.010334 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 18:27:20.010503 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.010340 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pkcf9\"" Oct 31 18:27:20.017921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.017903 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cgxbd"] Oct 31 18:27:20.026811 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.026783 5688 generic.go:356] "Generic (PLEG): container finished" podID="728c1b36-a297-4ff2-9a6b-c49816580ed6" containerID="fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074" exitCode=0 Oct 31 18:27:20.026936 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.026850 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" Oct 31 18:27:20.026936 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.026847 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" event={"ID":"728c1b36-a297-4ff2-9a6b-c49816580ed6","Type":"ContainerDied","Data":"fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074"} Oct 31 18:27:20.027052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.026967 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2" event={"ID":"728c1b36-a297-4ff2-9a6b-c49816580ed6","Type":"ContainerDied","Data":"286b6934caa8fa8dcd3d595f6c0371f53b3734f83c217e063054bdf4557ed37b"} Oct 31 18:27:20.027052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.026992 5688 scope.go:122] "RemoveContainer" containerID="fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074" Oct 31 18:27:20.035665 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.035647 5688 scope.go:122] "RemoveContainer" containerID="fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074" Oct 31 18:27:20.035923 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:27:20.035904 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074\": container with ID starting with fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074 not found: ID does not exist" containerID="fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074" Oct 31 18:27:20.035985 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.035937 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074"} err="failed to get container status \"fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074\": rpc error: code = NotFound desc = could not find container \"fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074\": container with ID starting with fae338d75b70b96eb7a9cd3122fee112ee0de40fc0e1837ea99289b58f5af074 not found: ID does not exist" Oct 31 18:27:20.055777 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.055758 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:20.098076 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.098047 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-metrics-certs\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.098244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.098094 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrbg\" (UniqueName: \"kubernetes.io/projected/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-kube-api-access-6qrbg\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.098244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.098178 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-cert\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.182146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.182061 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2"] Oct 31 18:27:20.184448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.184421 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xrvv2"] Oct 31 18:27:20.199089 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.199065 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-metrics-certs\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.199234 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.199102 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrbg\" (UniqueName: \"kubernetes.io/projected/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-kube-api-access-6qrbg\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.199295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.199230 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-cert\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.201600 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.201581 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-metrics-certs\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.201703 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.201615 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-cert\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.258308 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.258284 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrbg\" (UniqueName: \"kubernetes.io/projected/c75c1ba8-2b81-4a05-bf20-22aea0bdc837-kube-api-access-6qrbg\") pod \"odh-model-controller-769875db56-cgxbd\" (UID: \"c75c1ba8-2b81-4a05-bf20-22aea0bdc837\") " pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.269221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.269176 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:20.428146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.427996 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5"] Oct 31 18:27:20.430905 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:27:20.430871 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode4469264_59e8_491b_bbd4_dc5ca810caa5.slice/crio-aed0bbf0aed2b73f3733b772a7a3d3ed68bf63ddd84a7e60db689d71c6c25bc6 WatchSource:0}: Error finding container aed0bbf0aed2b73f3733b772a7a3d3ed68bf63ddd84a7e60db689d71c6c25bc6: Status 404 returned error can't find the container with id aed0bbf0aed2b73f3733b772a7a3d3ed68bf63ddd84a7e60db689d71c6c25bc6 Oct 31 18:27:20.653373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:20.653342 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cgxbd"] Oct 31 18:27:20.656695 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:27:20.656664 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75c1ba8_2b81_4a05_bf20_22aea0bdc837.slice/crio-9a525cd2bba5ca3a24ca1146ef192da562367263c3292df6799b423c8494736a WatchSource:0}: Error finding container 9a525cd2bba5ca3a24ca1146ef192da562367263c3292df6799b423c8494736a: Status 404 returned error can't find the container with id 9a525cd2bba5ca3a24ca1146ef192da562367263c3292df6799b423c8494736a Oct 31 18:27:21.032295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:21.032260 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" event={"ID":"e4469264-59e8-491b-bbd4-dc5ca810caa5","Type":"ContainerStarted","Data":"aed0bbf0aed2b73f3733b772a7a3d3ed68bf63ddd84a7e60db689d71c6c25bc6"} Oct 31 18:27:21.034149 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:21.034123 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cgxbd" event={"ID":"c75c1ba8-2b81-4a05-bf20-22aea0bdc837","Type":"ContainerStarted","Data":"9a525cd2bba5ca3a24ca1146ef192da562367263c3292df6799b423c8494736a"} Oct 31 18:27:21.566463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:21.566428 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728c1b36-a297-4ff2-9a6b-c49816580ed6" path="/var/lib/kubelet/pods/728c1b36-a297-4ff2-9a6b-c49816580ed6/volumes" Oct 31 18:27:24.050144 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:24.050104 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" event={"ID":"e4469264-59e8-491b-bbd4-dc5ca810caa5","Type":"ContainerStarted","Data":"c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b"} Oct 31 18:27:24.050619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:24.050177 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:24.051701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:24.051680 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cgxbd" event={"ID":"c75c1ba8-2b81-4a05-bf20-22aea0bdc837","Type":"ContainerStarted","Data":"1905848b733fdc26bb322ca3e95bdc4d77beaddde49c84ba996e95fb5280d555"} Oct 31 18:27:24.051782 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:24.051722 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:24.165312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:24.165212 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" podStartSLOduration=2.391906764 podStartE2EDuration="5.165171016s" totalImagesPullingTime="2.773264252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:24.16372895 +0000 UTC m=+364.765870094" watchObservedRunningTime="2026-10-31 18:27:24.165171016 +0000 UTC m=+364.767312157" Oct 31 18:27:24.277053 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:24.276998 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-cgxbd" podStartSLOduration=2.048864946 podStartE2EDuration="5.276983242s" totalImagesPullingTime="3.228118296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:24.275874112 +0000 UTC m=+364.878015256" watchObservedRunningTime="2026-10-31 18:27:24.276983242 +0000 UTC m=+364.879124382" Oct 31 18:27:24.903620 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:27:24.903582 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:27:35.058140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:35.058113 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-cgxbd" Oct 31 18:27:36.258954 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.258917 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl"] Oct 31 18:27:36.265785 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.265760 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.323631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.323603 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl"] Oct 31 18:27:36.372008 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.371981 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 18:27:36.372235 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.372218 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 18:27:36.372369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.372352 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\"" Oct 31 18:27:36.372421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.372407 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 18:27:36.372461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.372409 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 18:27:36.442527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.442497 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9kz\" (UniqueName: \"kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.442668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.442539 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.442668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.442610 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.442668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.442663 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.544172 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.544092 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9kz\" (UniqueName: \"kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.544172 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.544141 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.544172 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.544171 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.544410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.544216 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.544832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.544800 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.546660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.546641 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.546774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.546682 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.600782 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.600747 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9kz\" (UniqueName: \"kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz\") pod \"maas-api-6c7474d45f-kpwdl\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:36.878594 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:36.878514 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:37.252101 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:37.252070 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl"] Oct 31 18:27:37.255099 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:27:37.255073 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeec9d58_8ba2_45ee_a367_b9e685e93501.slice/crio-acc2bbedd00165deb61157079aaf9e614ecc1a52d09f4253226203028658a5ff WatchSource:0}: Error finding container acc2bbedd00165deb61157079aaf9e614ecc1a52d09f4253226203028658a5ff: Status 404 returned error can't find the container with id acc2bbedd00165deb61157079aaf9e614ecc1a52d09f4253226203028658a5ff Oct 31 18:27:38.112680 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:38.112645 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" event={"ID":"deec9d58-8ba2-45ee-a367-b9e685e93501","Type":"ContainerStarted","Data":"acc2bbedd00165deb61157079aaf9e614ecc1a52d09f4253226203028658a5ff"} Oct 31 18:27:43.138761 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:43.138724 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" event={"ID":"deec9d58-8ba2-45ee-a367-b9e685e93501","Type":"ContainerStarted","Data":"bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6"} Oct 31 18:27:43.139226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:43.138869 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:43.290078 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:43.289913 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" podStartSLOduration=1.530825208 podStartE2EDuration="7.289897996s" totalImagesPullingTime="5.759072788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:43.289229347 +0000 UTC m=+383.891370493" watchObservedRunningTime="2026-10-31 18:27:43.289897996 +0000 UTC m=+383.892039142" Oct 31 18:27:49.148135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:49.148103 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:50.172375 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.172343 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9645m"] Oct 31 18:27:50.175854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.175839 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.249750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.249724 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9645m"] Oct 31 18:27:50.258904 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.258878 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.259024 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.258986 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.259024 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.259012 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.259097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.259039 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.359605 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.359570 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.359775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.359614 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.359775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.359663 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.359775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.359742 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.360397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.360375 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.362086 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.362058 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.362281 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.362258 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.416373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.416351 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk\") pod \"maas-api-5cc786fc78-9645m\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.488350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.488325 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:50.859385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:50.859306 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9645m"] Oct 31 18:27:50.862297 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:27:50.862265 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5750a4_d7e3_42fa_be66_f86555616beb.slice/crio-9432f478b8a2ae9e3d7003da73492b9a4b1f73c614901192c50951d2eb744ff3 WatchSource:0}: Error finding container 9432f478b8a2ae9e3d7003da73492b9a4b1f73c614901192c50951d2eb744ff3: Status 404 returned error can't find the container with id 9432f478b8a2ae9e3d7003da73492b9a4b1f73c614901192c50951d2eb744ff3 Oct 31 18:27:51.171734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:51.171650 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" event={"ID":"5b5750a4-d7e3-42fa-be66-f86555616beb","Type":"ContainerStarted","Data":"9432f478b8a2ae9e3d7003da73492b9a4b1f73c614901192c50951d2eb744ff3"} Oct 31 18:27:52.177686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:52.177653 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" event={"ID":"5b5750a4-d7e3-42fa-be66-f86555616beb","Type":"ContainerStarted","Data":"dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db"} Oct 31 18:27:52.178050 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:52.177788 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:52.295477 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:52.295427 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" podStartSLOduration=1.45995084 podStartE2EDuration="2.29541254s" totalImagesPullingTime="835.4617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:52.294671546 +0000 UTC m=+392.896812690" watchObservedRunningTime="2026-10-31 18:27:52.29541254 +0000 UTC m=+392.897553678" Oct 31 18:27:54.905303 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:27:54.905264 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:27:55.057407 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:55.057381 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:27:58.187642 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.187616 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 18:27:58.361062 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.361024 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl"] Oct 31 18:27:58.361385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.361329 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" podUID="deec9d58-8ba2-45ee-a367-b9e685e93501" containerName="maas-api" containerID="cri-o://bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6" gracePeriod=30 Oct 31 18:27:58.602410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.602388 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:58.624761 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.624741 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca\") pod \"deec9d58-8ba2-45ee-a367-b9e685e93501\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " Oct 31 18:27:58.624838 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.624797 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz\" (UniqueName: \"kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz\") pod \"deec9d58-8ba2-45ee-a367-b9e685e93501\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " Oct 31 18:27:58.624883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.624845 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls\") pod \"deec9d58-8ba2-45ee-a367-b9e685e93501\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " Oct 31 18:27:58.624883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.624868 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs\") pod \"deec9d58-8ba2-45ee-a367-b9e685e93501\" (UID: \"deec9d58-8ba2-45ee-a367-b9e685e93501\") " Oct 31 18:27:58.625073 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.625043 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca" pod "deec9d58-8ba2-45ee-a367-b9e685e93501" (UID: "deec9d58-8ba2-45ee-a367-b9e685e93501"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:27:58.627033 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.627007 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs" pod "deec9d58-8ba2-45ee-a367-b9e685e93501" (UID: "deec9d58-8ba2-45ee-a367-b9e685e93501"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:58.627128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.627067 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls" pod "deec9d58-8ba2-45ee-a367-b9e685e93501" (UID: "deec9d58-8ba2-45ee-a367-b9e685e93501"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:27:58.627171 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.627146 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz" pod "deec9d58-8ba2-45ee-a367-b9e685e93501" (UID: "deec9d58-8ba2-45ee-a367-b9e685e93501"). InnerVolumeSpecName "kube-api-access-cz9kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:27:58.726226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.726177 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9kz\" (UniqueName: \"kubernetes.io/projected/deec9d58-8ba2-45ee-a367-b9e685e93501-kube-api-access-cz9kz\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:58.726340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.726229 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:58.726340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.726244 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deec9d58-8ba2-45ee-a367-b9e685e93501-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:58.726340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:58.726259 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec9d58-8ba2-45ee-a367-b9e685e93501-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:27:59.209211 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.209166 5688 generic.go:356] "Generic (PLEG): container finished" podID="deec9d58-8ba2-45ee-a367-b9e685e93501" containerID="bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6" exitCode=0 Oct 31 18:27:59.209604 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.209229 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" Oct 31 18:27:59.209604 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.209251 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" event={"ID":"deec9d58-8ba2-45ee-a367-b9e685e93501","Type":"ContainerDied","Data":"bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6"} Oct 31 18:27:59.209604 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.209305 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl" event={"ID":"deec9d58-8ba2-45ee-a367-b9e685e93501","Type":"ContainerDied","Data":"acc2bbedd00165deb61157079aaf9e614ecc1a52d09f4253226203028658a5ff"} Oct 31 18:27:59.209604 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.209327 5688 scope.go:122] "RemoveContainer" containerID="bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6" Oct 31 18:27:59.218463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.218448 5688 scope.go:122] "RemoveContainer" containerID="bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6" Oct 31 18:27:59.218691 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:27:59.218674 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6\": container with ID starting with bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6 not found: ID does not exist" containerID="bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6" Oct 31 18:27:59.218750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.218700 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6"} err="failed to get container status \"bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6\": rpc error: code = NotFound desc = could not find container \"bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6\": container with ID starting with bd9cc0913e022f24312eb19804f68a1f74fedd29e9de65f6aebb17265f9231a6 not found: ID does not exist" Oct 31 18:27:59.370928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.370894 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl"] Oct 31 18:27:59.420374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.420346 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c7474d45f-kpwdl"] Oct 31 18:27:59.563917 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:27:59.563894 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deec9d58-8ba2-45ee-a367-b9e685e93501" path="/var/lib/kubelet/pods/deec9d58-8ba2-45ee-a367-b9e685e93501/volumes" Oct 31 18:28:24.903764 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:28:24.903725 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:28:34.842046 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.842015 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-v99xr"] Oct 31 18:28:34.842512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.842362 5688 cpu_manager.go:414] "containerMap: removing container" podUID="deec9d58-8ba2-45ee-a367-b9e685e93501" containerName="maas-api" Oct 31 18:28:34.842512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.842374 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deec9d58-8ba2-45ee-a367-b9e685e93501" containerName="maas-api" Oct 31 18:28:34.842512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.842433 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="deec9d58-8ba2-45ee-a367-b9e685e93501" containerName="maas-api" Oct 31 18:28:34.845228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.845211 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:34.904762 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.904735 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-v99xr"] Oct 31 18:28:34.997663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.997631 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrpb\" (UniqueName: \"kubernetes.io/projected/17f0ac62-e7aa-4dea-8bae-dede767972e6-kube-api-access-brrpb\") pod \"llmisvc-controller-manager-6466c554d8-v99xr\" (UID: \"17f0ac62-e7aa-4dea-8bae-dede767972e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:34.997778 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:34.997684 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f0ac62-e7aa-4dea-8bae-dede767972e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-v99xr\" (UID: \"17f0ac62-e7aa-4dea-8bae-dede767972e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:35.099003 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:35.098938 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f0ac62-e7aa-4dea-8bae-dede767972e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-v99xr\" (UID: \"17f0ac62-e7aa-4dea-8bae-dede767972e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:35.099118 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:35.099019 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrpb\" (UniqueName: \"kubernetes.io/projected/17f0ac62-e7aa-4dea-8bae-dede767972e6-kube-api-access-brrpb\") pod \"llmisvc-controller-manager-6466c554d8-v99xr\" (UID: \"17f0ac62-e7aa-4dea-8bae-dede767972e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:35.101495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:35.101469 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f0ac62-e7aa-4dea-8bae-dede767972e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-v99xr\" (UID: \"17f0ac62-e7aa-4dea-8bae-dede767972e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:35.155799 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:35.155771 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrpb\" (UniqueName: \"kubernetes.io/projected/17f0ac62-e7aa-4dea-8bae-dede767972e6-kube-api-access-brrpb\") pod \"llmisvc-controller-manager-6466c554d8-v99xr\" (UID: \"17f0ac62-e7aa-4dea-8bae-dede767972e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:35.157574 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:35.157559 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:35.524705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:35.524679 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-v99xr"] Oct 31 18:28:35.527240 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:28:35.527214 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f0ac62_e7aa_4dea_8bae_dede767972e6.slice/crio-3f417a1584a04ca1c0088a14ae7c87d6b01fc9cd5160a2466d54a5aa12700171 WatchSource:0}: Error finding container 3f417a1584a04ca1c0088a14ae7c87d6b01fc9cd5160a2466d54a5aa12700171: Status 404 returned error can't find the container with id 3f417a1584a04ca1c0088a14ae7c87d6b01fc9cd5160a2466d54a5aa12700171 Oct 31 18:28:36.366075 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:36.366045 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" event={"ID":"17f0ac62-e7aa-4dea-8bae-dede767972e6","Type":"ContainerStarted","Data":"3f417a1584a04ca1c0088a14ae7c87d6b01fc9cd5160a2466d54a5aa12700171"} Oct 31 18:28:37.372712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:37.372668 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" event={"ID":"17f0ac62-e7aa-4dea-8bae-dede767972e6","Type":"ContainerStarted","Data":"ff629a13624e1e7e511d1e16c0df84dab192c0a4661442583c665a32e3c7e851"} Oct 31 18:28:37.373134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:37.372805 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:28:37.486723 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:28:37.486665 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" podStartSLOduration=2.704600513 podStartE2EDuration="3.486649144s" totalImagesPullingTime="782.048631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:28:37.485683538 +0000 UTC m=+438.087824681" watchObservedRunningTime="2026-10-31 18:28:37.486649144 +0000 UTC m=+438.088790285" Oct 31 18:28:54.903095 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:28:54.903058 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:29:08.378881 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.378811 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-v99xr" Oct 31 18:29:08.542550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.542517 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5"] Oct 31 18:29:08.542766 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.542733 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" podUID="e4469264-59e8-491b-bbd4-dc5ca810caa5" containerName="manager" containerID="cri-o://c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b" gracePeriod=30 Oct 31 18:29:08.785580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.785555 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:29:08.845473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.845447 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8\") pod \"e4469264-59e8-491b-bbd4-dc5ca810caa5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " Oct 31 18:29:08.845582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.845505 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert\" (UniqueName: \"kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert\") pod \"e4469264-59e8-491b-bbd4-dc5ca810caa5\" (UID: \"e4469264-59e8-491b-bbd4-dc5ca810caa5\") " Oct 31 18:29:08.847548 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.847514 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8" pod "e4469264-59e8-491b-bbd4-dc5ca810caa5" (UID: "e4469264-59e8-491b-bbd4-dc5ca810caa5"). InnerVolumeSpecName "kube-api-access-7czs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:29:08.847647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.847542 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert" pod "e4469264-59e8-491b-bbd4-dc5ca810caa5" (UID: "e4469264-59e8-491b-bbd4-dc5ca810caa5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:29:08.946421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.946362 5688 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4469264-59e8-491b-bbd4-dc5ca810caa5-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:29:08.946421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:08.946386 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/e4469264-59e8-491b-bbd4-dc5ca810caa5-kube-api-access-7czs8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:29:09.502493 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.502461 5688 generic.go:356] "Generic (PLEG): container finished" podID="e4469264-59e8-491b-bbd4-dc5ca810caa5" containerID="c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b" exitCode=0 Oct 31 18:29:09.502962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.502513 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" Oct 31 18:29:09.502962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.502543 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" event={"ID":"e4469264-59e8-491b-bbd4-dc5ca810caa5","Type":"ContainerDied","Data":"c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b"} Oct 31 18:29:09.502962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.502593 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5" event={"ID":"e4469264-59e8-491b-bbd4-dc5ca810caa5","Type":"ContainerDied","Data":"aed0bbf0aed2b73f3733b772a7a3d3ed68bf63ddd84a7e60db689d71c6c25bc6"} Oct 31 18:29:09.502962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.502614 5688 scope.go:122] "RemoveContainer" containerID="c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b" Oct 31 18:29:09.511593 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.511578 5688 scope.go:122] "RemoveContainer" containerID="c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b" Oct 31 18:29:09.511860 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:29:09.511841 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b\": container with ID starting with c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b not found: ID does not exist" containerID="c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b" Oct 31 18:29:09.511925 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.511872 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b"} err="failed to get container status \"c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b\": rpc error: code = NotFound desc = could not find container \"c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b\": container with ID starting with c5f6324dd67bba52ac12eeeccac6557953417a791791bf0153e9ffabd370fa4b not found: ID does not exist" Oct 31 18:29:09.760114 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.760043 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5"] Oct 31 18:29:09.764633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:09.764609 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2qfz5"] Oct 31 18:29:11.564809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:29:11.564777 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4469264-59e8-491b-bbd4-dc5ca810caa5" path="/var/lib/kubelet/pods/e4469264-59e8-491b-bbd4-dc5ca810caa5/volumes" Oct 31 18:29:24.903375 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:29:24.903342 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:29:54.902676 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:29:54.902631 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:30:24.903718 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:30:24.903682 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:30:47.453630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.453596 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5"] Oct 31 18:30:47.454030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.453915 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e4469264-59e8-491b-bbd4-dc5ca810caa5" containerName="manager" Oct 31 18:30:47.454030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.453925 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4469264-59e8-491b-bbd4-dc5ca810caa5" containerName="manager" Oct 31 18:30:47.454030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.453998 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4469264-59e8-491b-bbd4-dc5ca810caa5" containerName="manager" Oct 31 18:30:47.457012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.456993 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468679 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468659 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468687 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468710 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468727 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468860 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468831 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468866 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468886 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.468989 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468972 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.469022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.468999 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.537190 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.537153 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nsw28\"" Oct 31 18:30:47.565318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.565278 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5"] Oct 31 18:30:47.569706 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569678 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.569835 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569729 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.569835 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569759 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.569835 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569797 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.569835 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569823 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570104 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569893 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570104 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569919 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570104 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569962 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570104 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.569991 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.570298 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.570426 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.570558 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.570871 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.570801 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.571104 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.571078 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.572564 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.572543 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.572816 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.572730 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.626686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.626659 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.626794 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.626780 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lqzg5\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:47.769718 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:47.769695 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:48.191565 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:48.191497 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5"] Oct 31 18:30:48.194665 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:30:48.194634 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1093ffb_21b2_4373_89eb_a99ed8ea88b3.slice/crio-f5781ee8b88c3c03a5c3c70b764f76b62a48450c9d7cf7f8c575e702290352eb WatchSource:0}: Error finding container f5781ee8b88c3c03a5c3c70b764f76b62a48450c9d7cf7f8c575e702290352eb: Status 404 returned error can't find the container with id f5781ee8b88c3c03a5c3c70b764f76b62a48450c9d7cf7f8c575e702290352eb Oct 31 18:30:48.903582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:48.903551 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" event={"ID":"f1093ffb-21b2-4373-89eb-a99ed8ea88b3","Type":"ContainerStarted","Data":"f5781ee8b88c3c03a5c3c70b764f76b62a48450c9d7cf7f8c575e702290352eb"} Oct 31 18:30:52.766491 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:52.766449 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:30:52.766753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:52.766534 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:30:52.766753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:52.766562 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:30:52.924846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:52.924809 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" event={"ID":"f1093ffb-21b2-4373-89eb-a99ed8ea88b3","Type":"ContainerStarted","Data":"6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10"} Oct 31 18:30:53.041280 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:53.041132 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podStartSLOduration=1.472023267 podStartE2EDuration="6.041117786s" totalImagesPullingTime="4.569094519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:53.040348541 +0000 UTC m=+573.642489695" watchObservedRunningTime="2026-10-31 18:30:53.041117786 +0000 UTC m=+573.643258929" Oct 31 18:30:53.770671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:53.770634 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:30:53.772240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:53.772217 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 18:30:53.772364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:53.772258 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 18:30:54.770438 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:54.770402 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 18:30:54.770675 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:54.770463 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 18:30:54.903980 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:30:54.903940 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:30:55.770344 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:55.770314 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 18:30:55.770505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:55.770378 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 18:30:56.770844 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:56.770810 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 18:30:56.771342 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:56.770883 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 18:30:57.770740 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:57.770707 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 18:30:57.770916 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:57.770760 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 18:30:58.770467 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:58.770429 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 18:30:58.770657 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:30:58.770485 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 18:31:00.488617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:00.488575 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:31:00.489128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:00.489108 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:31:00.489931 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:00.489916 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 18:31:02.585088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.585060 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs"] Oct 31 18:31:02.588354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.588338 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615493 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615471 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615599 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615509 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615651 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615594 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615651 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615621 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfb8\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615676 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615773 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615726 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615773 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615748 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615848 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615831 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.615885 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.615858 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.685674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.685645 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs"] Oct 31 18:31:02.716162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716138 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716175 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716218 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716234 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716253 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716269 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716296 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716312 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfb8\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716405 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716627 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716747 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716689 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.716796 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.716772 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.717109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.717092 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.717202 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.717120 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.718677 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.718653 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.718899 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.718878 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.772950 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.772922 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.773035 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.772949 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfb8\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9p8qs\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:02.901396 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:02.901342 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:03.322931 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:03.322897 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs"] Oct 31 18:31:03.325910 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:31:03.325884 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cafa0b_8ca9_4b39_ba41_4931964674f8.slice/crio-f12e2ca24d3a851aae1f3f64c47948a4dd8cd774b5d953a5ccb392b62dad7ad1 WatchSource:0}: Error finding container f12e2ca24d3a851aae1f3f64c47948a4dd8cd774b5d953a5ccb392b62dad7ad1: Status 404 returned error can't find the container with id f12e2ca24d3a851aae1f3f64c47948a4dd8cd774b5d953a5ccb392b62dad7ad1 Oct 31 18:31:03.328764 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:03.328731 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:31:03.328870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:03.328806 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:31:03.328870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:03.328847 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:31:03.993739 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:03.993707 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" event={"ID":"56cafa0b-8ca9-4b39-ba41-4931964674f8","Type":"ContainerStarted","Data":"e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478"} Oct 31 18:31:03.994141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:03.993764 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" event={"ID":"56cafa0b-8ca9-4b39-ba41-4931964674f8","Type":"ContainerStarted","Data":"f12e2ca24d3a851aae1f3f64c47948a4dd8cd774b5d953a5ccb392b62dad7ad1"} Oct 31 18:31:04.902232 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:04.902199 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:04.903632 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:04.903609 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 18:31:04.903742 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:04.903665 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 18:31:05.901689 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:05.901651 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 18:31:05.902072 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:05.901711 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 18:31:06.902358 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:06.902322 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 18:31:06.902729 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:06.902381 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 18:31:07.901949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:07.901915 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 18:31:07.902127 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:07.901968 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 18:31:08.901904 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:08.901872 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 18:31:08.902310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:08.901923 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 18:31:09.902427 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:09.902397 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 18:31:09.902792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:09.902439 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 18:31:11.229670 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:11.229640 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:11.230152 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:11.230103 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:11.230888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:11.230864 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:31:11.448002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:11.447960 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podStartSLOduration=9.447948618 podStartE2EDuration="9.447948618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:04.153912545 +0000 UTC m=+584.756053690" watchObservedRunningTime="2026-10-31 18:31:11.447948618 +0000 UTC m=+592.050089761" Oct 31 18:31:19.611085 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:19.611058 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:31:19.611944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:19.611923 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:31:24.903337 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:31:24.903301 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:31:26.688519 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.688486 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp"] Oct 31 18:31:26.695297 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.695277 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.798856 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.798826 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e4c133ca-serving-cert\"" Oct 31 18:31:26.800120 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.800101 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp"] Oct 31 18:31:26.856077 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.856044 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.856239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.856082 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54qm\" (UniqueName: \"kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.856239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.856197 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.856239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.856234 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.957022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.956950 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.957022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.956986 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54qm\" (UniqueName: \"kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.957179 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.957023 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.957179 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.957057 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.957774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.957748 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.959508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.959483 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:26.959644 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:26.959625 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:27.018155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:27.018134 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54qm\" (UniqueName: \"kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm\") pod \"maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:27.307748 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:27.307720 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:27.676572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:27.676501 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp"] Oct 31 18:31:27.679736 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:31:27.679706 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab8b715_5c4c_454a_8315_e3beb3977fd7.slice/crio-f7daaa87d3416f969ff79af71cb2fb0bdc9ca15d6508fd4c0f77dea3624cc85b WatchSource:0}: Error finding container f7daaa87d3416f969ff79af71cb2fb0bdc9ca15d6508fd4c0f77dea3624cc85b: Status 404 returned error can't find the container with id f7daaa87d3416f969ff79af71cb2fb0bdc9ca15d6508fd4c0f77dea3624cc85b Oct 31 18:31:28.097545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:28.097507 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" event={"ID":"5ab8b715-5c4c-454a-8315-e3beb3977fd7","Type":"ContainerStarted","Data":"f7daaa87d3416f969ff79af71cb2fb0bdc9ca15d6508fd4c0f77dea3624cc85b"} Oct 31 18:31:29.103934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:29.103899 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" event={"ID":"5ab8b715-5c4c-454a-8315-e3beb3977fd7","Type":"ContainerStarted","Data":"ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8"} Oct 31 18:31:29.104381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:29.104056 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:29.217301 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:29.217245 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" podStartSLOduration=2.366527874 podStartE2EDuration="3.217227372s" totalImagesPullingTime="850.699498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:29.216589317 +0000 UTC m=+609.818730463" watchObservedRunningTime="2026-10-31 18:31:29.217227372 +0000 UTC m=+609.819368514" Oct 31 18:31:31.930167 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:31.930133 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp"] Oct 31 18:31:31.933668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:31.933647 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:31.985678 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:31.985652 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bbf5caaf-serving-cert\"" Oct 31 18:31:32.036712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.036687 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp"] Oct 31 18:31:32.102543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.102511 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.102677 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.102568 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcz8\" (UniqueName: \"kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.102728 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.102707 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.102781 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.102767 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.203642 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.203547 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.203642 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.203622 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.203851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.203664 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.203851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.203692 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcz8\" (UniqueName: \"kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.204375 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.204340 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.206121 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.206102 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.206274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.206253 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.260244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.260214 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcz8\" (UniqueName: \"kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8\") pod \"maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.545324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.545292 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:32.921233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:32.921119 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp"] Oct 31 18:31:32.925173 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:31:32.925145 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9975e939_9cb8_45c3_aee3_d7c3f9ac393f.slice/crio-bb23e7528f83bd85bc578c368cd7ea13ec6f1671eee14321fbd4eac2fd8f5a0a WatchSource:0}: Error finding container bb23e7528f83bd85bc578c368cd7ea13ec6f1671eee14321fbd4eac2fd8f5a0a: Status 404 returned error can't find the container with id bb23e7528f83bd85bc578c368cd7ea13ec6f1671eee14321fbd4eac2fd8f5a0a Oct 31 18:31:33.125222 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:33.125170 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" event={"ID":"9975e939-9cb8-45c3-aee3-d7c3f9ac393f","Type":"ContainerStarted","Data":"bb23e7528f83bd85bc578c368cd7ea13ec6f1671eee14321fbd4eac2fd8f5a0a"} Oct 31 18:31:34.131498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:34.131422 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" event={"ID":"9975e939-9cb8-45c3-aee3-d7c3f9ac393f","Type":"ContainerStarted","Data":"bcedf67b93824e20a5a66e602d9cee60afe9d7aad4fbe21f40c6ee84fbd57896"} Oct 31 18:31:34.131864 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:34.131519 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:34.246672 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:34.246621 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" podStartSLOduration=2.349988608 podStartE2EDuration="3.246607222s" totalImagesPullingTime="896.618614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:34.245440019 +0000 UTC m=+614.847581169" watchObservedRunningTime="2026-10-31 18:31:34.246607222 +0000 UTC m=+614.848748368" Oct 31 18:31:35.114170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:35.114138 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:31:37.108175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.108135 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws"] Oct 31 18:31:37.111517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.111502 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.164509 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.164479 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1f4b473d-serving-cert\"" Oct 31 18:31:37.214968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.214942 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws"] Oct 31 18:31:37.246696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.246667 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.246822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.246711 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.246893 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.246871 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.247002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.246985 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxth\" (UniqueName: \"kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.348465 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.348443 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.348607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.348471 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.348607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.348504 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.348607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.348547 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxth\" (UniqueName: \"kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.348778 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:31:37.348624 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-serving-cert: secret "maas-api-e2e-migrate-1f4b473d-serving-cert" not found Oct 31 18:31:37.348778 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:31:37.348717 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls podName:024183b5-9898-4f25-97d9-8a9246b4e087 nodeName:}" failed. No retries permitted until 2026-10-31 18:31:37.848695487 +0000 UTC m=+618.450836613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls") pod "maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" (UID: "024183b5-9898-4f25-97d9-8a9246b4e087") : secret "maas-api-e2e-migrate-1f4b473d-serving-cert" not found Oct 31 18:31:37.349083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.349062 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.352055 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.352012 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.406148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.406083 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxth\" (UniqueName: \"kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.857246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.857210 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:37.859617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:37.859598 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\") pod \"maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:38.023243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:38.023215 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:38.394451 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:38.394404 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws"] Oct 31 18:31:38.397061 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:31:38.397030 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024183b5_9898_4f25_97d9_8a9246b4e087.slice/crio-7a919eea5cd211dd108bb2a01978111800f70318e511e514d04aba8f1e73e779 WatchSource:0}: Error finding container 7a919eea5cd211dd108bb2a01978111800f70318e511e514d04aba8f1e73e779: Status 404 returned error can't find the container with id 7a919eea5cd211dd108bb2a01978111800f70318e511e514d04aba8f1e73e779 Oct 31 18:31:39.154385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:39.154344 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" event={"ID":"024183b5-9898-4f25-97d9-8a9246b4e087","Type":"ContainerStarted","Data":"7a919eea5cd211dd108bb2a01978111800f70318e511e514d04aba8f1e73e779"} Oct 31 18:31:40.139836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:40.139810 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:31:40.160115 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:40.160088 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" event={"ID":"024183b5-9898-4f25-97d9-8a9246b4e087","Type":"ContainerStarted","Data":"0be35235b9e464bfda45d1f1a52ca701c2840b2d424e3921fc92e7147d9d6a7d"} Oct 31 18:31:40.160275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:40.160233 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:40.373029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:40.372980 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" podStartSLOduration=2.5478773009999998 podStartE2EDuration="3.372964903s" totalImagesPullingTime="825.087602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:40.372556101 +0000 UTC m=+620.974697236" watchObservedRunningTime="2026-10-31 18:31:40.372964903 +0000 UTC m=+620.975106040" Oct 31 18:31:46.170386 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:31:46.170355 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:31:54.904835 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:31:54.904799 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:32:20.725858 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:20.725764 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp"] Oct 31 18:32:20.726458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:20.725999 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" podUID="5ab8b715-5c4c-454a-8315-e3beb3977fd7" containerName="maas-api" containerID="cri-o://ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8" gracePeriod=30 Oct 31 18:32:20.984762 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:20.984741 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:32:21.124254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.123237 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs\") pod \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " Oct 31 18:32:21.124254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.123361 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm\" (UniqueName: \"kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm\") pod \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " Oct 31 18:32:21.124254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.123393 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls\") pod \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " Oct 31 18:32:21.124254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.123428 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca\") pod \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\" (UID: \"5ab8b715-5c4c-454a-8315-e3beb3977fd7\") " Oct 31 18:32:21.124643 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.124496 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca" pod "5ab8b715-5c4c-454a-8315-e3beb3977fd7" (UID: "5ab8b715-5c4c-454a-8315-e3beb3977fd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:32:21.127146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.127118 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs" pod "5ab8b715-5c4c-454a-8315-e3beb3977fd7" (UID: "5ab8b715-5c4c-454a-8315-e3beb3977fd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:32:21.128335 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.128305 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm" pod "5ab8b715-5c4c-454a-8315-e3beb3977fd7" (UID: "5ab8b715-5c4c-454a-8315-e3beb3977fd7"). InnerVolumeSpecName "kube-api-access-f54qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:32:21.128571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.128535 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls" pod "5ab8b715-5c4c-454a-8315-e3beb3977fd7" (UID: "5ab8b715-5c4c-454a-8315-e3beb3977fd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:32:21.234233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.230218 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f54qm\" (UniqueName: \"kubernetes.io/projected/5ab8b715-5c4c-454a-8315-e3beb3977fd7-kube-api-access-f54qm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:21.234233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.230252 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:21.234233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.230270 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8b715-5c4c-454a-8315-e3beb3977fd7-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:21.234233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.230284 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab8b715-5c4c-454a-8315-e3beb3977fd7-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:21.334350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.334268 5688 generic.go:356] "Generic (PLEG): container finished" podID="5ab8b715-5c4c-454a-8315-e3beb3977fd7" containerID="ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8" exitCode=0 Oct 31 18:32:21.334350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.334313 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" Oct 31 18:32:21.334350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.334339 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" event={"ID":"5ab8b715-5c4c-454a-8315-e3beb3977fd7","Type":"ContainerDied","Data":"ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8"} Oct 31 18:32:21.334598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.334391 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp" event={"ID":"5ab8b715-5c4c-454a-8315-e3beb3977fd7","Type":"ContainerDied","Data":"f7daaa87d3416f969ff79af71cb2fb0bdc9ca15d6508fd4c0f77dea3624cc85b"} Oct 31 18:32:21.334598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.334418 5688 scope.go:122] "RemoveContainer" containerID="ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8" Oct 31 18:32:21.343008 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.342993 5688 scope.go:122] "RemoveContainer" containerID="ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8" Oct 31 18:32:21.343263 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:32:21.343244 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8\": container with ID starting with ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8 not found: ID does not exist" containerID="ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8" Oct 31 18:32:21.343340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.343277 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8"} err="failed to get container status \"ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8\": rpc error: code = NotFound desc = could not find container \"ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8\": container with ID starting with ff3bb656885082f1c2c980932630a62bcfe6918d12221dcde9fd2640f59dccc8 not found: ID does not exist" Oct 31 18:32:21.496806 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.496785 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp"] Oct 31 18:32:21.501012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.500990 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4c133ca-56c9db666c-w4fcp"] Oct 31 18:32:21.565853 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:21.565830 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab8b715-5c4c-454a-8315-e3beb3977fd7" path="/var/lib/kubelet/pods/5ab8b715-5c4c-454a-8315-e3beb3977fd7/volumes" Oct 31 18:32:24.902955 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:32:24.902920 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:32:27.081944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.081912 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp"] Oct 31 18:32:27.082546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.082157 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" podUID="9975e939-9cb8-45c3-aee3-d7c3f9ac393f" containerName="maas-api" containerID="cri-o://bcedf67b93824e20a5a66e602d9cee60afe9d7aad4fbe21f40c6ee84fbd57896" gracePeriod=30 Oct 31 18:32:27.362955 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.362925 5688 generic.go:356] "Generic (PLEG): container finished" podID="9975e939-9cb8-45c3-aee3-d7c3f9ac393f" containerID="bcedf67b93824e20a5a66e602d9cee60afe9d7aad4fbe21f40c6ee84fbd57896" exitCode=0 Oct 31 18:32:27.363092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.362968 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" event={"ID":"9975e939-9cb8-45c3-aee3-d7c3f9ac393f","Type":"ContainerDied","Data":"bcedf67b93824e20a5a66e602d9cee60afe9d7aad4fbe21f40c6ee84fbd57896"} Oct 31 18:32:27.380434 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.380410 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:32:27.482037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.482000 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls\") pod \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " Oct 31 18:32:27.482261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.482056 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8\" (UniqueName: \"kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8\") pod \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " Oct 31 18:32:27.482261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.482113 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca\") pod \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " Oct 31 18:32:27.482261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.482130 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs\") pod \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\" (UID: \"9975e939-9cb8-45c3-aee3-d7c3f9ac393f\") " Oct 31 18:32:27.482567 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.482517 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca" pod "9975e939-9cb8-45c3-aee3-d7c3f9ac393f" (UID: "9975e939-9cb8-45c3-aee3-d7c3f9ac393f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:32:27.484410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.484378 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs" pod "9975e939-9cb8-45c3-aee3-d7c3f9ac393f" (UID: "9975e939-9cb8-45c3-aee3-d7c3f9ac393f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:32:27.484410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.484387 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8" pod "9975e939-9cb8-45c3-aee3-d7c3f9ac393f" (UID: "9975e939-9cb8-45c3-aee3-d7c3f9ac393f"). InnerVolumeSpecName "kube-api-access-rfcz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:32:27.484571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.484555 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls" pod "9975e939-9cb8-45c3-aee3-d7c3f9ac393f" (UID: "9975e939-9cb8-45c3-aee3-d7c3f9ac393f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:32:27.583674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.583648 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:27.583674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.583672 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfcz8\" (UniqueName: \"kubernetes.io/projected/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-kube-api-access-rfcz8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:27.583819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.583681 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:27.583819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:27.583689 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9975e939-9cb8-45c3-aee3-d7c3f9ac393f-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:28.223877 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.223845 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws"] Oct 31 18:32:28.224293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.224016 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" podUID="024183b5-9898-4f25-97d9-8a9246b4e087" containerName="maas-api" containerID="cri-o://0be35235b9e464bfda45d1f1a52ca701c2840b2d424e3921fc92e7147d9d6a7d" gracePeriod=30 Oct 31 18:32:28.371063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.371029 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" event={"ID":"9975e939-9cb8-45c3-aee3-d7c3f9ac393f","Type":"ContainerDied","Data":"bb23e7528f83bd85bc578c368cd7ea13ec6f1671eee14321fbd4eac2fd8f5a0a"} Oct 31 18:32:28.371275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.371045 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp" Oct 31 18:32:28.371275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.371128 5688 scope.go:122] "RemoveContainer" containerID="bcedf67b93824e20a5a66e602d9cee60afe9d7aad4fbe21f40c6ee84fbd57896" Oct 31 18:32:28.373038 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.372932 5688 generic.go:356] "Generic (PLEG): container finished" podID="024183b5-9898-4f25-97d9-8a9246b4e087" containerID="0be35235b9e464bfda45d1f1a52ca701c2840b2d424e3921fc92e7147d9d6a7d" exitCode=0 Oct 31 18:32:28.373038 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.372968 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" event={"ID":"024183b5-9898-4f25-97d9-8a9246b4e087","Type":"ContainerDied","Data":"0be35235b9e464bfda45d1f1a52ca701c2840b2d424e3921fc92e7147d9d6a7d"} Oct 31 18:32:28.478553 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.478489 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:32:28.536582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.536552 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp"] Oct 31 18:32:28.539684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.539658 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf5caaf-69759b8854-9cbhp"] Oct 31 18:32:28.591203 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.591155 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth\" (UniqueName: \"kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth\") pod \"024183b5-9898-4f25-97d9-8a9246b4e087\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " Oct 31 18:32:28.591311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.591240 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca\") pod \"024183b5-9898-4f25-97d9-8a9246b4e087\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " Oct 31 18:32:28.591350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.591317 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\") pod \"024183b5-9898-4f25-97d9-8a9246b4e087\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " Oct 31 18:32:28.591387 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.591352 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs\") pod \"024183b5-9898-4f25-97d9-8a9246b4e087\" (UID: \"024183b5-9898-4f25-97d9-8a9246b4e087\") " Oct 31 18:32:28.591570 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.591551 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca" pod "024183b5-9898-4f25-97d9-8a9246b4e087" (UID: "024183b5-9898-4f25-97d9-8a9246b4e087"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:32:28.593464 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.593434 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth" pod "024183b5-9898-4f25-97d9-8a9246b4e087" (UID: "024183b5-9898-4f25-97d9-8a9246b4e087"). InnerVolumeSpecName "kube-api-access-4qxth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:32:28.593555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.593481 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs" pod "024183b5-9898-4f25-97d9-8a9246b4e087" (UID: "024183b5-9898-4f25-97d9-8a9246b4e087"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:32:28.593555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.593497 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls" pod "024183b5-9898-4f25-97d9-8a9246b4e087" (UID: "024183b5-9898-4f25-97d9-8a9246b4e087"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:32:28.692457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.692426 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxth\" (UniqueName: \"kubernetes.io/projected/024183b5-9898-4f25-97d9-8a9246b4e087-kube-api-access-4qxth\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:28.692457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.692456 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024183b5-9898-4f25-97d9-8a9246b4e087-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:28.692659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.692472 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:28.692659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:28.692484 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024183b5-9898-4f25-97d9-8a9246b4e087-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:32:29.379106 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:29.379076 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" Oct 31 18:32:29.379586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:29.379081 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws" event={"ID":"024183b5-9898-4f25-97d9-8a9246b4e087","Type":"ContainerDied","Data":"7a919eea5cd211dd108bb2a01978111800f70318e511e514d04aba8f1e73e779"} Oct 31 18:32:29.379586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:29.379251 5688 scope.go:122] "RemoveContainer" containerID="0be35235b9e464bfda45d1f1a52ca701c2840b2d424e3921fc92e7147d9d6a7d" Oct 31 18:32:29.540512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:29.540484 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws"] Oct 31 18:32:29.565103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:29.565078 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9975e939-9cb8-45c3-aee3-d7c3f9ac393f" path="/var/lib/kubelet/pods/9975e939-9cb8-45c3-aee3-d7c3f9ac393f/volumes" Oct 31 18:32:29.589440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:29.589407 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f4b473d-fdfcf7fdb-qgrws"] Oct 31 18:32:31.563937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:31.563909 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024183b5-9898-4f25-97d9-8a9246b4e087" path="/var/lib/kubelet/pods/024183b5-9898-4f25-97d9-8a9246b4e087/volumes" Oct 31 18:32:32.104685 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.104654 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc"] Oct 31 18:32:32.105062 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105047 5688 cpu_manager.go:414] "containerMap: removing container" podUID="024183b5-9898-4f25-97d9-8a9246b4e087" containerName="maas-api" Oct 31 18:32:32.105062 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105062 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024183b5-9898-4f25-97d9-8a9246b4e087" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105073 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5ab8b715-5c4c-454a-8315-e3beb3977fd7" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105077 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab8b715-5c4c-454a-8315-e3beb3977fd7" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105088 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9975e939-9cb8-45c3-aee3-d7c3f9ac393f" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105095 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9975e939-9cb8-45c3-aee3-d7c3f9ac393f" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105148 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="024183b5-9898-4f25-97d9-8a9246b4e087" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105155 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab8b715-5c4c-454a-8315-e3beb3977fd7" containerName="maas-api" Oct 31 18:32:32.105175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.105161 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9975e939-9cb8-45c3-aee3-d7c3f9ac393f" containerName="maas-api" Oct 31 18:32:32.108223 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.108205 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.164387 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.164342 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6279f5-openshift-default-dockercfg-d8cq6\"" Oct 31 18:32:32.215107 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.215065 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc"] Oct 31 18:32:32.226283 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226256 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226294 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226311 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226331 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226506 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226403 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226506 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226466 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226536 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9b8s\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226558 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.226637 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.226593 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327070 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327122 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327147 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327162 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327179 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327212 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327418 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327493 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9b8s\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327528 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327647 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327693 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.327953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.327934 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.328021 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.328001 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.328259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.328239 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.330473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.330442 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.330735 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.330711 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.384601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.384531 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.384750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.384728 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9b8s\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.421631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.421600 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:32.847131 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.847090 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc"] Oct 31 18:32:32.849895 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:32:32.849865 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e00ec3_6702_4512_9022_c7e604564478.slice/crio-1c02753696ecac543f3984cd2968b2f1d550556e9d98b8c54b0d324883ef05d4 WatchSource:0}: Error finding container 1c02753696ecac543f3984cd2968b2f1d550556e9d98b8c54b0d324883ef05d4: Status 404 returned error can't find the container with id 1c02753696ecac543f3984cd2968b2f1d550556e9d98b8c54b0d324883ef05d4 Oct 31 18:32:32.851921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.851888 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:32:32.851990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.851950 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:32:32.851990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:32.851978 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:32:33.400571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:33.400533 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" event={"ID":"70e00ec3-6702-4512-9022-c7e604564478","Type":"ContainerStarted","Data":"b21e278dcec06a5c2cd6fa0a724841b452153533181873a5981595331d563d04"} Oct 31 18:32:33.400764 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:33.400587 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" event={"ID":"70e00ec3-6702-4512-9022-c7e604564478","Type":"ContainerStarted","Data":"1c02753696ecac543f3984cd2968b2f1d550556e9d98b8c54b0d324883ef05d4"} Oct 31 18:32:33.422574 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:33.422549 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:33.427369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:33.427350 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:33.517920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:33.517043 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podStartSLOduration=1.517023391 podStartE2EDuration="1.517023391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:33.516531445 +0000 UTC m=+674.118672588" watchObservedRunningTime="2026-10-31 18:32:33.517023391 +0000 UTC m=+674.119164534" Oct 31 18:32:34.404529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:34.404489 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:34.405566 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:32:34.405542 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:32:54.903878 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:32:54.903837 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:33:12.601271 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.601205 5688 generic.go:356] "Generic (PLEG): container finished" podID="70e00ec3-6702-4512-9022-c7e604564478" containerID="b21e278dcec06a5c2cd6fa0a724841b452153533181873a5981595331d563d04" exitCode=0 Oct 31 18:33:12.601659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.601269 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" event={"ID":"70e00ec3-6702-4512-9022-c7e604564478","Type":"ContainerDied","Data":"b21e278dcec06a5c2cd6fa0a724841b452153533181873a5981595331d563d04"} Oct 31 18:33:12.601659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.601577 5688 scope.go:122] "RemoveContainer" containerID="b21e278dcec06a5c2cd6fa0a724841b452153533181873a5981595331d563d04" Oct 31 18:33:12.602725 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.602697 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:33:12.602785 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.602764 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:33:12.602822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.602795 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:33:12.964888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.964796 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr"] Oct 31 18:33:12.975927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:12.975898 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.015720 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.015688 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr"] Oct 31 18:33:13.017866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.017835 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.018026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.017893 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.018026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.017925 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.018026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.017951 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.018026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.017995 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwbp\" (UniqueName: \"kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.018256 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.018030 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.028716 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.028662 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"default-dockercfg-5cdvq\"" Oct 31 18:33:13.028879 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.028733 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"kube-root-ca.crt\"" Oct 31 18:33:13.028879 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.028745 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"openshift-service-ca.crt\"" Oct 31 18:33:13.028879 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.028810 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"simulator-w2-88085e-kserve-self-signed-certs\"" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119497 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119564 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119597 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119625 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119676 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwbp\" (UniqueName: \"kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119730 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.119996 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.120400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.120353 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.121913 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.121788 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.124307 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.124285 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.125794 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.125777 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.183253 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.183216 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwbp\" (UniqueName: \"kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp\") pod \"simulator-w2-88085e-kserve-797bd57d75-stnkr\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.308213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.308162 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:13.607964 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.607884 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" event={"ID":"70e00ec3-6702-4512-9022-c7e604564478","Type":"ContainerStarted","Data":"9836adb9ac24cfa7d8377ace8bdc81a3a73c0048c3020456c249faf24d687830"} Oct 31 18:33:13.695250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.695217 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr"] Oct 31 18:33:13.705793 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:33:13.705750 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6e10e4_0667_4d3f_996f_32b87a52b5de.slice/crio-5f610f2dc9f861962916115a678af2e98b3d7879d057a9f50e962a0f8fd4b374 WatchSource:0}: Error finding container 5f610f2dc9f861962916115a678af2e98b3d7879d057a9f50e962a0f8fd4b374: Status 404 returned error can't find the container with id 5f610f2dc9f861962916115a678af2e98b3d7879d057a9f50e962a0f8fd4b374 Oct 31 18:33:13.708303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:13.708269 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 18:33:14.422546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:14.422516 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:33:14.422792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:14.422705 5688 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 18:33:14.422904 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:14.422815 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 18:33:14.623292 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:14.623254 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" event={"ID":"2b6e10e4-0667-4d3f-996f-32b87a52b5de","Type":"ContainerStarted","Data":"5f610f2dc9f861962916115a678af2e98b3d7879d057a9f50e962a0f8fd4b374"} Oct 31 18:33:15.422595 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:15.422557 5688 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 18:33:15.422773 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:15.422623 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 18:33:16.422866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:16.422831 5688 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 18:33:16.423266 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:16.422888 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 18:33:16.658066 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:16.658018 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" event={"ID":"2b6e10e4-0667-4d3f-996f-32b87a52b5de","Type":"ContainerStarted","Data":"4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987"} Oct 31 18:33:16.773832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:16.773778 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" podStartSLOduration=2.855626873 podStartE2EDuration="4.773763734s" totalImagesPullingTime="1.918136861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:16.772023138 +0000 UTC m=+717.374164283" watchObservedRunningTime="2026-10-31 18:33:16.773763734 +0000 UTC m=+717.375904878" Oct 31 18:33:17.308305 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:17.308268 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:17.422518 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:17.422484 5688 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 18:33:17.422684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:17.422548 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 18:33:18.422505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:18.422467 5688 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 18:33:18.422941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:18.422531 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 18:33:19.405459 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:19.405414 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:33:19.426057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:19.426031 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:33:19.673041 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:19.672979 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 18:33:23.308341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:23.308311 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:23.319934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:23.319912 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:23.701818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:23.701739 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 18:33:24.904609 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:33:24.904575 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:33:37.248651 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.248606 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h"] Oct 31 18:33:37.251992 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.251977 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.352034 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.352003 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"kube-root-ca.crt\"" Oct 31 18:33:37.356625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.356604 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"openshift-service-ca.crt\"" Oct 31 18:33:37.356789 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.356770 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"simulator-w4-6279f5-kserve-self-signed-certs\"" Oct 31 18:33:37.356842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.356821 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"default-dockercfg-hwl4n\"" Oct 31 18:33:37.359522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.359505 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h"] Oct 31 18:33:37.361654 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.361637 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.361701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.361665 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.361701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.361690 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.361849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.361830 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.361898 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.361862 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.361990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.361974 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn664\" (UniqueName: \"kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463446 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463411 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn664\" (UniqueName: \"kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463457 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463478 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463495 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463568 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463593 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463903 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.463987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463933 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.464047 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.463981 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.466018 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.465993 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.466219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.466205 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.523581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.523514 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn664\" (UniqueName: \"kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664\") pod \"simulator-w4-6279f5-kserve-796c58b6b4-pp85h\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.563444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.563257 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:37.943441 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:37.943374 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h"] Oct 31 18:33:37.946692 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:33:37.946659 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dde1a57_750a_4b21_9530_8e9917cf6b3a.slice/crio-70bb715292823a5f985f8ac4d193aebf848179418f476b8e51240a4fa257f8e1 WatchSource:0}: Error finding container 70bb715292823a5f985f8ac4d193aebf848179418f476b8e51240a4fa257f8e1: Status 404 returned error can't find the container with id 70bb715292823a5f985f8ac4d193aebf848179418f476b8e51240a4fa257f8e1 Oct 31 18:33:38.753447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:38.753413 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" event={"ID":"0dde1a57-750a-4b21-9530-8e9917cf6b3a","Type":"ContainerStarted","Data":"dc80df2beeb35082da40d5142811b0454796eaa52f78c1621771da3e8053667a"} Oct 31 18:33:38.753447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:38.753453 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" event={"ID":"0dde1a57-750a-4b21-9530-8e9917cf6b3a","Type":"ContainerStarted","Data":"70bb715292823a5f985f8ac4d193aebf848179418f476b8e51240a4fa257f8e1"} Oct 31 18:33:38.870067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:38.870018 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" podStartSLOduration=1.8700038239999999 podStartE2EDuration="1.870003824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:38.868440109 +0000 UTC m=+739.470581253" watchObservedRunningTime="2026-10-31 18:33:38.870003824 +0000 UTC m=+739.472144968" Oct 31 18:33:39.572472 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:39.572443 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:47.564379 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:47.564348 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:47.575669 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:47.575644 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:47.809083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:47.809060 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:33:54.904010 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:33:54.903977 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:33:56.661103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.661069 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x"] Oct 31 18:33:56.664633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.664613 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.766209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.766160 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"premium-w2-88085e-kserve-self-signed-certs\"" Oct 31 18:33:56.767550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.767532 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x"] Oct 31 18:33:56.827817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.827782 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.828009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.827841 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.828158 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.828129 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.828312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.828279 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.828397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.828330 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.828397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.828372 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnktv\" (UniqueName: \"kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929114 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929148 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnktv\" (UniqueName: \"kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929168 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929232 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929299 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929354 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929723 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929848 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929761 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.929848 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.929777 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.932147 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.932115 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.932301 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.932286 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:56.985599 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:56.985571 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnktv\" (UniqueName: \"kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv\") pod \"premium-w2-88085e-kserve-d68487646-gbs8x\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:57.276393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:57.276362 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:33:57.645134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:57.645037 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x"] Oct 31 18:33:57.647695 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:33:57.647670 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dabdcc5_d3c9_4eb6_a396_941b73b9d3ad.slice/crio-ba1ccbe059d62a8845dfa212bf5779be719bccbd7390ffc27222eccf2a7474d9 WatchSource:0}: Error finding container ba1ccbe059d62a8845dfa212bf5779be719bccbd7390ffc27222eccf2a7474d9: Status 404 returned error can't find the container with id ba1ccbe059d62a8845dfa212bf5779be719bccbd7390ffc27222eccf2a7474d9 Oct 31 18:33:57.849006 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:57.848974 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" event={"ID":"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad","Type":"ContainerStarted","Data":"8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33"} Oct 31 18:33:57.849394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:57.849015 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" event={"ID":"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad","Type":"ContainerStarted","Data":"ba1ccbe059d62a8845dfa212bf5779be719bccbd7390ffc27222eccf2a7474d9"} Oct 31 18:33:57.962598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:57.962513 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" podStartSLOduration=1.962500634 podStartE2EDuration="1.962500634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:57.961080109 +0000 UTC m=+758.563221254" watchObservedRunningTime="2026-10-31 18:33:57.962500634 +0000 UTC m=+758.564641778" Oct 31 18:33:58.276525 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:33:58.276490 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:34:05.671485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.671453 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv"] Oct 31 18:34:05.674702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.674685 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.727447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.727421 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"openshift-service-ca.crt\"" Oct 31 18:34:05.727447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.727421 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"default-dockercfg-dchgm\"" Oct 31 18:34:05.727447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.727421 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"kube-root-ca.crt\"" Oct 31 18:34:05.727650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.727425 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"simulator-w1-da6ea7-kserve-self-signed-certs\"" Oct 31 18:34:05.733812 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.733788 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv"] Oct 31 18:34:05.821453 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.821418 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.821638 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.821464 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.821638 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.821500 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.821753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.821635 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.821753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.821687 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.821753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.821739 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhxq\" (UniqueName: \"kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.922663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922584 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.922663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922619 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.922663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922639 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.922663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922664 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.922970 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922689 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.922970 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922727 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhxq\" (UniqueName: \"kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.923135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.922968 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.923135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.923080 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.923252 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.923132 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.925489 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.925468 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.925853 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.925835 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.979742 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.979715 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhxq\" (UniqueName: \"kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq\") pod \"simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:05.986593 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:05.986567 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:06.361708 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:06.361673 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv"] Oct 31 18:34:06.364576 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:34:06.364555 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c52dce0_0ca5_4bf2_a7c6_a62efe325f14.slice/crio-74f4aea3ad559e16f6bd36a3d524d17b023b03291907e3090908142b97d3a38e WatchSource:0}: Error finding container 74f4aea3ad559e16f6bd36a3d524d17b023b03291907e3090908142b97d3a38e: Status 404 returned error can't find the container with id 74f4aea3ad559e16f6bd36a3d524d17b023b03291907e3090908142b97d3a38e Oct 31 18:34:06.891336 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:06.891304 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" event={"ID":"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14","Type":"ContainerStarted","Data":"bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970"} Oct 31 18:34:06.891336 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:06.891342 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" event={"ID":"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14","Type":"ContainerStarted","Data":"74f4aea3ad559e16f6bd36a3d524d17b023b03291907e3090908142b97d3a38e"} Oct 31 18:34:06.987179 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:06.987153 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:07.007924 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:07.007882 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podStartSLOduration=2.007869559 podStartE2EDuration="2.007869559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:34:07.007087185 +0000 UTC m=+767.609228330" watchObservedRunningTime="2026-10-31 18:34:07.007869559 +0000 UTC m=+767.610010703" Oct 31 18:34:07.276914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:07.276883 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:34:07.288734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:07.288705 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:34:07.906271 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:07.906235 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:34:15.990222 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:15.987938 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:16.003163 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:16.003132 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:16.948263 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:16.948236 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:34:24.904900 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:34:24.904859 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:34:29.934001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:29.933971 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94"] Oct 31 18:34:29.938593 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:29.938573 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.026823 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.026786 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.026972 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.026844 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.026972 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.026950 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjt5\" (UniqueName: \"kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.027070 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.026991 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.027143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.027128 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.027227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.027217 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.041661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.041636 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"premium-w4-6279f5-kserve-self-signed-certs\"" Oct 31 18:34:30.042853 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.042836 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94"] Oct 31 18:34:30.127907 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.127869 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.128201 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.128155 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.128334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.128257 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.128334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.128298 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.128445 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.128354 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.128445 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.128399 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjt5\" (UniqueName: \"kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.128739 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.128713 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.129107 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.129086 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.129472 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.129453 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.131211 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.131166 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.131780 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.131762 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.185480 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.185412 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjt5\" (UniqueName: \"kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5\") pod \"premium-w4-6279f5-kserve-76d79fc69c-t2d94\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.250928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.250890 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:30.660649 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.660617 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94"] Oct 31 18:34:30.664253 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:34:30.664221 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641de09a_8b10_4de2_a672_e0a751b1c646.slice/crio-71ed91a792011181cc14a9abfd7ba05316bb4a58cb6635afc6bb083315f4f4fd WatchSource:0}: Error finding container 71ed91a792011181cc14a9abfd7ba05316bb4a58cb6635afc6bb083315f4f4fd: Status 404 returned error can't find the container with id 71ed91a792011181cc14a9abfd7ba05316bb4a58cb6635afc6bb083315f4f4fd Oct 31 18:34:30.997169 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.997139 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" event={"ID":"641de09a-8b10-4de2-a672-e0a751b1c646","Type":"ContainerStarted","Data":"6e36144e0b7dc5ba7344d1e696e1f62c8e048842d7c4de803fb3547c6ea38a04"} Oct 31 18:34:30.997582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:30.997206 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" event={"ID":"641de09a-8b10-4de2-a672-e0a751b1c646","Type":"ContainerStarted","Data":"71ed91a792011181cc14a9abfd7ba05316bb4a58cb6635afc6bb083315f4f4fd"} Oct 31 18:34:31.109674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:31.109627 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" podStartSLOduration=2.109612871 podStartE2EDuration="2.109612871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:34:31.108135483 +0000 UTC m=+791.710276626" watchObservedRunningTime="2026-10-31 18:34:31.109612871 +0000 UTC m=+791.711754014" Oct 31 18:34:31.251278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:31.251202 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:40.251362 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:40.251322 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:40.263007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:40.262980 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:41.051936 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:34:41.051907 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:34:54.903751 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:34:54.903715 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:35:08.545850 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.545781 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms"] Oct 31 18:35:08.549479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.549460 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.651044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.651022 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"premium-w1-da6ea7-kserve-self-signed-certs\"" Oct 31 18:35:08.652516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.652496 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms"] Oct 31 18:35:08.664718 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.664697 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.664814 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.664728 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bst\" (UniqueName: \"kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.664814 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.664751 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.664942 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.664919 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.665027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.665009 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.665061 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.665049 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766114 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766148 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bst\" (UniqueName: \"kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766172 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766226 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766271 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766306 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766606 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766677 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766641 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.766819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.766794 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.768749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.768727 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.769011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.768992 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.822687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.822622 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bst\" (UniqueName: \"kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst\") pod \"premium-w1-da6ea7-kserve-75874d5dd7-smpms\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:08.879539 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:08.879504 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:09.278103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:09.278071 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms"] Oct 31 18:35:09.281467 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:35:09.281437 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83b91cf0_7975_4c70_bc07_807330d477ef.slice/crio-b605c617347c5673efbeff6f73d826283b2365362d6df1caf88416348160ce60 WatchSource:0}: Error finding container b605c617347c5673efbeff6f73d826283b2365362d6df1caf88416348160ce60: Status 404 returned error can't find the container with id b605c617347c5673efbeff6f73d826283b2365362d6df1caf88416348160ce60 Oct 31 18:35:10.181933 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:10.181896 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" event={"ID":"83b91cf0-7975-4c70-bc07-807330d477ef","Type":"ContainerStarted","Data":"77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031"} Oct 31 18:35:10.182429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:10.181940 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" event={"ID":"83b91cf0-7975-4c70-bc07-807330d477ef","Type":"ContainerStarted","Data":"b605c617347c5673efbeff6f73d826283b2365362d6df1caf88416348160ce60"} Oct 31 18:35:10.295130 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:10.295082 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" podStartSLOduration=2.295069193 podStartE2EDuration="2.295069193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:35:10.293694122 +0000 UTC m=+830.895835266" watchObservedRunningTime="2026-10-31 18:35:10.295069193 +0000 UTC m=+830.897210337" Oct 31 18:35:10.880434 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:10.880401 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:18.880021 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:18.879986 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:18.893582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:18.893553 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:19.233410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:19.233384 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:35:24.905049 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:35:24.905014 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:35:31.877948 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.877916 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q"] Oct 31 18:35:31.881478 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.881462 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:31.971560 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.971531 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2l4n\" (UniqueName: \"kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:31.971707 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.971585 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:31.971707 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.971689 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:31.983098 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.983075 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:35:31.983252 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.983233 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:35:31.984585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:31.984564 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q"] Oct 31 18:35:32.072131 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.072103 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.072289 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.072158 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l4n\" (UniqueName: \"kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.072289 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.072229 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.073033 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.072957 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.074650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.074630 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.130588 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.130528 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l4n\" (UniqueName: \"kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n\") pod \"maas-api-revoke-sub-7e12ee39-7cc5q\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.194156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.194135 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:32.564127 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:32.564099 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q"] Oct 31 18:35:32.567291 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:35:32.567266 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f3d2d0_4bd2_4420_8dac_294ba3caebd2.slice/crio-734d7d7b815f912eac24581dec4f6777031311384763ba49115ad74a857f074e WatchSource:0}: Error finding container 734d7d7b815f912eac24581dec4f6777031311384763ba49115ad74a857f074e: Status 404 returned error can't find the container with id 734d7d7b815f912eac24581dec4f6777031311384763ba49115ad74a857f074e Oct 31 18:35:33.287983 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:33.287950 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" event={"ID":"04f3d2d0-4bd2-4420-8dac-294ba3caebd2","Type":"ContainerStarted","Data":"734d7d7b815f912eac24581dec4f6777031311384763ba49115ad74a857f074e"} Oct 31 18:35:37.325024 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:37.324977 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" event={"ID":"04f3d2d0-4bd2-4420-8dac-294ba3caebd2","Type":"ContainerStarted","Data":"9ec7c2dbc45dfc850c11b0115a1f8bb4c129555faa03e2e99a9e7a2f280430d6"} Oct 31 18:35:37.440698 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:37.440649 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" podStartSLOduration=2.5255481250000003 podStartE2EDuration="6.44063689s" totalImagesPullingTime="3.915088765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:35:37.439371251 +0000 UTC m=+858.041512396" watchObservedRunningTime="2026-10-31 18:35:37.44063689 +0000 UTC m=+858.042778034" Oct 31 18:35:38.330941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:38.330911 5688 generic.go:356] "Generic (PLEG): container finished" podID="04f3d2d0-4bd2-4420-8dac-294ba3caebd2" containerID="9ec7c2dbc45dfc850c11b0115a1f8bb4c129555faa03e2e99a9e7a2f280430d6" exitCode=0 Oct 31 18:35:38.331345 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:38.330948 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" event={"ID":"04f3d2d0-4bd2-4420-8dac-294ba3caebd2","Type":"ContainerDied","Data":"9ec7c2dbc45dfc850c11b0115a1f8bb4c129555faa03e2e99a9e7a2f280430d6"} Oct 31 18:35:39.471160 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.471131 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:39.548012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.547985 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n\" (UniqueName: \"kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n\") pod \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " Oct 31 18:35:39.548123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.548026 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca\") pod \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " Oct 31 18:35:39.548165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.548123 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca\") pod \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\" (UID: \"04f3d2d0-4bd2-4420-8dac-294ba3caebd2\") " Oct 31 18:35:39.548476 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.548450 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca" pod "04f3d2d0-4bd2-4420-8dac-294ba3caebd2" (UID: "04f3d2d0-4bd2-4420-8dac-294ba3caebd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:35:39.550093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.550071 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca" pod "04f3d2d0-4bd2-4420-8dac-294ba3caebd2" (UID: "04f3d2d0-4bd2-4420-8dac-294ba3caebd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:35:39.550170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.550134 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n" pod "04f3d2d0-4bd2-4420-8dac-294ba3caebd2" (UID: "04f3d2d0-4bd2-4420-8dac-294ba3caebd2"). InnerVolumeSpecName "kube-api-access-l2l4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:35:39.648808 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.648744 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:35:39.648808 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.648770 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2l4n\" (UniqueName: \"kubernetes.io/projected/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-kube-api-access-l2l4n\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:35:39.648808 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:39.648779 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f3d2d0-4bd2-4420-8dac-294ba3caebd2-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:35:40.341011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:40.340978 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" event={"ID":"04f3d2d0-4bd2-4420-8dac-294ba3caebd2","Type":"ContainerDied","Data":"734d7d7b815f912eac24581dec4f6777031311384763ba49115ad74a857f074e"} Oct 31 18:35:40.341011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:40.341006 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q" Oct 31 18:35:40.341230 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:40.341031 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734d7d7b815f912eac24581dec4f6777031311384763ba49115ad74a857f074e" Oct 31 18:35:45.235542 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.235505 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22"] Oct 31 18:35:45.236139 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.236119 5688 cpu_manager.go:414] "containerMap: removing container" podUID="04f3d2d0-4bd2-4420-8dac-294ba3caebd2" containerName="revoke-keys" Oct 31 18:35:45.236208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.236142 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f3d2d0-4bd2-4420-8dac-294ba3caebd2" containerName="revoke-keys" Oct 31 18:35:45.236275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.236263 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f3d2d0-4bd2-4420-8dac-294ba3caebd2" containerName="revoke-keys" Oct 31 18:35:45.239777 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.239758 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.343380 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.343332 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"openshift-service-ca.crt\"" Oct 31 18:35:45.343544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.343399 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"kube-root-ca.crt\"" Oct 31 18:35:45.343544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.343478 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"premium-w0-a8587d-kserve-self-signed-certs\"" Oct 31 18:35:45.343651 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.343634 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"default-dockercfg-dmdhf\"" Oct 31 18:35:45.344946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.344924 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22"] Oct 31 18:35:45.401497 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.401455 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.401694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.401511 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.401694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.401569 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.401694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.401611 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js47t\" (UniqueName: \"kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.401694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.401673 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.401953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.401699 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502414 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502452 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502499 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502538 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js47t\" (UniqueName: \"kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502592 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502626 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.502910 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.502890 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.503036 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.503012 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.503259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.503238 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.505495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.505466 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.505653 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.505636 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.560212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.560166 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js47t\" (UniqueName: \"kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t\") pod \"premium-w0-a8587d-kserve-f9fcc6555-whj22\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:45.851563 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:45.851486 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:46.223667 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:46.223598 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22"] Oct 31 18:35:46.227036 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:35:46.227005 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod603d0d27_08f1_4cf0_8979_462837af1f29.slice/crio-1e1951549e1fc7a295c5f99adcbd014fb17fa326348b26c74f05b37ec38d999f WatchSource:0}: Error finding container 1e1951549e1fc7a295c5f99adcbd014fb17fa326348b26c74f05b37ec38d999f: Status 404 returned error can't find the container with id 1e1951549e1fc7a295c5f99adcbd014fb17fa326348b26c74f05b37ec38d999f Oct 31 18:35:46.371385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:46.371352 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" event={"ID":"603d0d27-08f1-4cf0-8979-462837af1f29","Type":"ContainerStarted","Data":"499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99"} Oct 31 18:35:46.371385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:46.371389 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" event={"ID":"603d0d27-08f1-4cf0-8979-462837af1f29","Type":"ContainerStarted","Data":"1e1951549e1fc7a295c5f99adcbd014fb17fa326348b26c74f05b37ec38d999f"} Oct 31 18:35:46.530837 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:46.530789 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" podStartSLOduration=1.5307766329999999 podStartE2EDuration="1.530776633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:35:46.530391583 +0000 UTC m=+867.132532763" watchObservedRunningTime="2026-10-31 18:35:46.530776633 +0000 UTC m=+867.132917773" Oct 31 18:35:46.851683 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:46.851613 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:49.669818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.669783 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt"] Oct 31 18:35:49.673479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.673454 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.775135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.775107 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:35:49.775135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.775107 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:35:49.776669 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.776632 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt"] Oct 31 18:35:49.843338 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.843311 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.843507 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.843342 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.843507 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.843364 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc28h\" (UniqueName: \"kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.944655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.944576 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.944655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.944613 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.944655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.944632 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc28h\" (UniqueName: \"kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.945440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.945413 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:49.947002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:49.946973 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:50.001287 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:50.001260 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc28h\" (UniqueName: \"kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h\") pod \"maas-api-revoke-sub-7d7443e1-c78xt\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:50.286757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:50.286732 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:50.655739 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:50.655665 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt"] Oct 31 18:35:50.659058 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:35:50.659032 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5452e03a_8fcf_4f18_ba50_783fad74bc0a.slice/crio-a39d134e506603225a785dd310fb599d79ee8751ecf69ef23b64c25a81636d2f WatchSource:0}: Error finding container a39d134e506603225a785dd310fb599d79ee8751ecf69ef23b64c25a81636d2f: Status 404 returned error can't find the container with id a39d134e506603225a785dd310fb599d79ee8751ecf69ef23b64c25a81636d2f Oct 31 18:35:51.398485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:51.398448 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" event={"ID":"5452e03a-8fcf-4f18-ba50-783fad74bc0a","Type":"ContainerStarted","Data":"d473ce31682d2ca6c2df6a502593454176a1db155d49fbfa9203b38536a855e1"} Oct 31 18:35:51.398872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:51.398506 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" event={"ID":"5452e03a-8fcf-4f18-ba50-783fad74bc0a","Type":"ContainerStarted","Data":"a39d134e506603225a785dd310fb599d79ee8751ecf69ef23b64c25a81636d2f"} Oct 31 18:35:51.510379 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:51.510336 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" podStartSLOduration=2.510321251 podStartE2EDuration="2.510321251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:35:51.508605499 +0000 UTC m=+872.110746644" watchObservedRunningTime="2026-10-31 18:35:51.510321251 +0000 UTC m=+872.112462427" Oct 31 18:35:52.405991 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:52.405957 5688 generic.go:356] "Generic (PLEG): container finished" podID="5452e03a-8fcf-4f18-ba50-783fad74bc0a" containerID="d473ce31682d2ca6c2df6a502593454176a1db155d49fbfa9203b38536a855e1" exitCode=0 Oct 31 18:35:52.406406 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:52.406027 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" event={"ID":"5452e03a-8fcf-4f18-ba50-783fad74bc0a","Type":"ContainerDied","Data":"d473ce31682d2ca6c2df6a502593454176a1db155d49fbfa9203b38536a855e1"} Oct 31 18:35:53.548960 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.548937 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:53.682805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.682736 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca\") pod \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " Oct 31 18:35:53.682962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.682898 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca\") pod \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " Oct 31 18:35:53.682962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.682942 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h\" (UniqueName: \"kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h\") pod \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\" (UID: \"5452e03a-8fcf-4f18-ba50-783fad74bc0a\") " Oct 31 18:35:53.683168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.683139 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca" pod "5452e03a-8fcf-4f18-ba50-783fad74bc0a" (UID: "5452e03a-8fcf-4f18-ba50-783fad74bc0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:35:53.683435 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.683404 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:35:53.685665 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.685629 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h" pod "5452e03a-8fcf-4f18-ba50-783fad74bc0a" (UID: "5452e03a-8fcf-4f18-ba50-783fad74bc0a"). InnerVolumeSpecName "kube-api-access-wc28h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:35:53.686348 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.686322 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca" pod "5452e03a-8fcf-4f18-ba50-783fad74bc0a" (UID: "5452e03a-8fcf-4f18-ba50-783fad74bc0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:35:53.784296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.784268 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5452e03a-8fcf-4f18-ba50-783fad74bc0a-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:35:53.784296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:53.784292 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc28h\" (UniqueName: \"kubernetes.io/projected/5452e03a-8fcf-4f18-ba50-783fad74bc0a-kube-api-access-wc28h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:35:54.416808 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:54.416775 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" event={"ID":"5452e03a-8fcf-4f18-ba50-783fad74bc0a","Type":"ContainerDied","Data":"a39d134e506603225a785dd310fb599d79ee8751ecf69ef23b64c25a81636d2f"} Oct 31 18:35:54.417000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:54.416813 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt" Oct 31 18:35:54.417000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:54.416820 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a39d134e506603225a785dd310fb599d79ee8751ecf69ef23b64c25a81636d2f" Oct 31 18:35:54.904573 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:35:54.904528 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:35:55.851901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:55.851872 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:55.864255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:55.864231 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:35:56.437561 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:35:56.437531 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:36:03.754430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.754398 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h"] Oct 31 18:36:03.754872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.754760 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5452e03a-8fcf-4f18-ba50-783fad74bc0a" containerName="revoke-keys" Oct 31 18:36:03.754872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.754771 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5452e03a-8fcf-4f18-ba50-783fad74bc0a" containerName="revoke-keys" Oct 31 18:36:03.754872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.754841 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5452e03a-8fcf-4f18-ba50-783fad74bc0a" containerName="revoke-keys" Oct 31 18:36:03.757624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.757606 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.772415 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.772393 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.772512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.772496 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.772555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.772523 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4m7\" (UniqueName: \"kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.815500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.815471 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h"] Oct 31 18:36:03.859375 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.859342 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:36:03.859492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.859388 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:36:03.873732 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.873695 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4m7\" (UniqueName: \"kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.873871 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.873822 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.873932 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.873922 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.874677 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.874652 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.876362 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.876338 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:03.929753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:03.929732 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4m7\" (UniqueName: \"kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7\") pod \"maas-api-revoke-sub-0bb83ce8-wzd5h\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:04.070248 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:04.070162 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:04.441444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:04.441368 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h"] Oct 31 18:36:04.444477 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:36:04.444448 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5d8c02_c661_41e5_bdd4_8bfd52b1d0cc.slice/crio-bc81f1ba1352cbf97a13be3f54c2c1008cc9c73d8ee436e19691b9094107beee WatchSource:0}: Error finding container bc81f1ba1352cbf97a13be3f54c2c1008cc9c73d8ee436e19691b9094107beee: Status 404 returned error can't find the container with id bc81f1ba1352cbf97a13be3f54c2c1008cc9c73d8ee436e19691b9094107beee Oct 31 18:36:04.465698 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:04.465671 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" event={"ID":"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc","Type":"ContainerStarted","Data":"bc81f1ba1352cbf97a13be3f54c2c1008cc9c73d8ee436e19691b9094107beee"} Oct 31 18:36:05.472103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:05.472031 5688 generic.go:356] "Generic (PLEG): container finished" podID="5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" containerID="7e0befc608172d4dfcd7490256b4e280e846f975a5592b84e018ad8d86ce6c18" exitCode=0 Oct 31 18:36:05.472103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:05.472066 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" event={"ID":"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc","Type":"ContainerDied","Data":"7e0befc608172d4dfcd7490256b4e280e846f975a5592b84e018ad8d86ce6c18"} Oct 31 18:36:06.616653 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.616626 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:06.698999 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.698966 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca\") pod \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " Oct 31 18:36:06.699141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.699038 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca\") pod \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " Oct 31 18:36:06.699141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.699104 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7\" (UniqueName: \"kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7\") pod \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\" (UID: \"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc\") " Oct 31 18:36:06.699333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.699306 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca" pod "5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" (UID: "5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:36:06.699475 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.699455 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:06.701106 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.701082 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca" pod "5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" (UID: "5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:36:06.701314 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.701291 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7" pod "5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" (UID: "5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc"). InnerVolumeSpecName "kube-api-access-rl4m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:36:06.800520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.800487 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:06.800520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:06.800511 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl4m7\" (UniqueName: \"kubernetes.io/projected/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc-kube-api-access-rl4m7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:07.481949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:07.481916 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" event={"ID":"5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc","Type":"ContainerDied","Data":"bc81f1ba1352cbf97a13be3f54c2c1008cc9c73d8ee436e19691b9094107beee"} Oct 31 18:36:07.481949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:07.481938 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h" Oct 31 18:36:07.482211 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:07.481985 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc81f1ba1352cbf97a13be3f54c2c1008cc9c73d8ee436e19691b9094107beee" Oct 31 18:36:19.299854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.299820 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc"] Oct 31 18:36:19.300329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.300173 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" containerName="revoke-keys" Oct 31 18:36:19.300329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.300197 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" containerName="revoke-keys" Oct 31 18:36:19.300329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.300293 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" containerName="revoke-keys" Oct 31 18:36:19.306878 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.306862 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.360953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.360924 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc"] Oct 31 18:36:19.408564 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.408538 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:36:19.408678 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.408598 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:36:19.408827 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.408809 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.408863 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.408847 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42p9n\" (UniqueName: \"kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.408929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.408916 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.509819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.509792 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.509944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.509873 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.509944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.509908 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42p9n\" (UniqueName: \"kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.510555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.510536 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.560968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.560908 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:36:19.567077 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.567054 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42p9n\" (UniqueName: \"kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.572847 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.572826 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103a940f-xmlgc\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:19.640530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.640512 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:36:19.643445 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.643423 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:36:19.677091 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.676874 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:36:19.679658 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:19.679635 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:20.053105 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:20.053079 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc"] Oct 31 18:36:20.056372 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:36:20.056342 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf240f5_a75e_47a9_8a8d_e60e0b6e7acb.slice/crio-f76649d262e193179bde6bd37c936a0b6329180bf39b861746dcda009e535a67 WatchSource:0}: Error finding container f76649d262e193179bde6bd37c936a0b6329180bf39b861746dcda009e535a67: Status 404 returned error can't find the container with id f76649d262e193179bde6bd37c936a0b6329180bf39b861746dcda009e535a67 Oct 31 18:36:20.544499 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:20.544463 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" event={"ID":"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb","Type":"ContainerStarted","Data":"484617f66d33932e8aff067e91c4f8788dfbfa4daa6439674518d184edae9181"} Oct 31 18:36:20.544857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:20.544523 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" event={"ID":"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb","Type":"ContainerStarted","Data":"f76649d262e193179bde6bd37c936a0b6329180bf39b861746dcda009e535a67"} Oct 31 18:36:20.657869 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:20.657706 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" podStartSLOduration=1.6576892079999999 podStartE2EDuration="1.657689208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:20.657009059 +0000 UTC m=+901.259150204" watchObservedRunningTime="2026-10-31 18:36:20.657689208 +0000 UTC m=+901.259830355" Oct 31 18:36:21.550432 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:21.550395 5688 generic.go:356] "Generic (PLEG): container finished" podID="1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" containerID="484617f66d33932e8aff067e91c4f8788dfbfa4daa6439674518d184edae9181" exitCode=0 Oct 31 18:36:21.550798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:21.550484 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" event={"ID":"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb","Type":"ContainerDied","Data":"484617f66d33932e8aff067e91c4f8788dfbfa4daa6439674518d184edae9181"} Oct 31 18:36:22.692520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.692496 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:22.740860 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.740828 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca\") pod \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " Oct 31 18:36:22.740971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.740959 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n\" (UniqueName: \"kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n\") pod \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " Oct 31 18:36:22.741025 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.740986 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca\") pod \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\" (UID: \"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb\") " Oct 31 18:36:22.741390 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.741358 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca" pod "1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" (UID: "1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:36:22.742843 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.742816 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca" pod "1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" (UID: "1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:36:22.743067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.743047 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n" pod "1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" (UID: "1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb"). InnerVolumeSpecName "kube-api-access-42p9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:36:22.842099 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.842041 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:22.842099 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.842064 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42p9n\" (UniqueName: \"kubernetes.io/projected/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-kube-api-access-42p9n\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:22.842099 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:22.842074 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:23.563946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:23.563917 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" Oct 31 18:36:23.565978 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:23.565944 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc" event={"ID":"1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb","Type":"ContainerDied","Data":"f76649d262e193179bde6bd37c936a0b6329180bf39b861746dcda009e535a67"} Oct 31 18:36:23.566114 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:23.565986 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76649d262e193179bde6bd37c936a0b6329180bf39b861746dcda009e535a67" Oct 31 18:36:24.904112 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:36:24.904072 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:36:25.665868 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.665840 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm"] Oct 31 18:36:25.666221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.666208 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" containerName="revoke-keys" Oct 31 18:36:25.666277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.666222 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" containerName="revoke-keys" Oct 31 18:36:25.666321 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.666296 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" containerName="revoke-keys" Oct 31 18:36:25.669787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.669763 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.724676 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.724650 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:36:25.764296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.764269 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.764460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.764343 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.764538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.764487 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvkg\" (UniqueName: \"kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.774259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.774229 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:36:25.775866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.775848 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm"] Oct 31 18:36:25.865555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.865529 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.865687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.865625 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvkg\" (UniqueName: \"kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.865687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.865663 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.866314 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.866297 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.867947 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.867931 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.922106 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.922048 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvkg\" (UniqueName: \"kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg\") pod \"maas-api-revoke-sub-c08b7312-gxxcm\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:25.984351 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:25.984324 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:26.353264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:26.353122 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm"] Oct 31 18:36:26.356047 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:36:26.356019 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4708701b_35ae_469f_bad0_63cbf0417e14.slice/crio-277709e3666e179d4fd829d56ca78464cefb96462b68881a828b064b29da3dce WatchSource:0}: Error finding container 277709e3666e179d4fd829d56ca78464cefb96462b68881a828b064b29da3dce: Status 404 returned error can't find the container with id 277709e3666e179d4fd829d56ca78464cefb96462b68881a828b064b29da3dce Oct 31 18:36:26.578088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:26.578059 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" event={"ID":"4708701b-35ae-469f-bad0-63cbf0417e14","Type":"ContainerStarted","Data":"277709e3666e179d4fd829d56ca78464cefb96462b68881a828b064b29da3dce"} Oct 31 18:36:27.583571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:27.583540 5688 generic.go:356] "Generic (PLEG): container finished" podID="4708701b-35ae-469f-bad0-63cbf0417e14" containerID="4216dad6d281377bd99347c96381d37d9ef2ded92cc25b5bf346d8596532b7e9" exitCode=0 Oct 31 18:36:27.583938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:27.583626 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" event={"ID":"4708701b-35ae-469f-bad0-63cbf0417e14","Type":"ContainerDied","Data":"4216dad6d281377bd99347c96381d37d9ef2ded92cc25b5bf346d8596532b7e9"} Oct 31 18:36:28.726279 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.726255 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:28.797215 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.797169 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca\") pod \"4708701b-35ae-469f-bad0-63cbf0417e14\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " Oct 31 18:36:28.797358 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.797227 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg\" (UniqueName: \"kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg\") pod \"4708701b-35ae-469f-bad0-63cbf0417e14\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " Oct 31 18:36:28.797358 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.797254 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca\") pod \"4708701b-35ae-469f-bad0-63cbf0417e14\" (UID: \"4708701b-35ae-469f-bad0-63cbf0417e14\") " Oct 31 18:36:28.797630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.797609 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca" pod "4708701b-35ae-469f-bad0-63cbf0417e14" (UID: "4708701b-35ae-469f-bad0-63cbf0417e14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:36:28.799383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.799361 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg" pod "4708701b-35ae-469f-bad0-63cbf0417e14" (UID: "4708701b-35ae-469f-bad0-63cbf0417e14"). InnerVolumeSpecName "kube-api-access-hmvkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:36:28.799383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.799369 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca" pod "4708701b-35ae-469f-bad0-63cbf0417e14" (UID: "4708701b-35ae-469f-bad0-63cbf0417e14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:36:28.898559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.898501 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:28.898559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.898526 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmvkg\" (UniqueName: \"kubernetes.io/projected/4708701b-35ae-469f-bad0-63cbf0417e14-kube-api-access-hmvkg\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:28.898559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:28.898537 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4708701b-35ae-469f-bad0-63cbf0417e14-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:29.594166 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:29.594135 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" event={"ID":"4708701b-35ae-469f-bad0-63cbf0417e14","Type":"ContainerDied","Data":"277709e3666e179d4fd829d56ca78464cefb96462b68881a828b064b29da3dce"} Oct 31 18:36:29.594327 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:29.594152 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm" Oct 31 18:36:29.594327 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:29.594213 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277709e3666e179d4fd829d56ca78464cefb96462b68881a828b064b29da3dce" Oct 31 18:36:36.443081 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.443042 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59"] Oct 31 18:36:36.443928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.443887 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4708701b-35ae-469f-bad0-63cbf0417e14" containerName="revoke-keys" Oct 31 18:36:36.443928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.443907 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4708701b-35ae-469f-bad0-63cbf0417e14" containerName="revoke-keys" Oct 31 18:36:36.444164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.444147 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4708701b-35ae-469f-bad0-63cbf0417e14" containerName="revoke-keys" Oct 31 18:36:36.448068 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.448047 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.502635 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.502605 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59"] Oct 31 18:36:36.550073 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.550044 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:36:36.550224 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.550092 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:36:36.563883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.563859 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.563988 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.563911 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.564027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.563983 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsn8r\" (UniqueName: \"kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.665249 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.665221 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.665579 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.665526 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.665787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.665767 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsn8r\" (UniqueName: \"kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.666361 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.666329 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.667921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.667904 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.722063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.722011 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsn8r\" (UniqueName: \"kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r\") pod \"maas-api-revoke-sub-376095ff-flt59\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:36.760832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:36.760808 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:37.140337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:37.140309 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59"] Oct 31 18:36:37.142787 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:36:37.142759 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dc18d1_6681_4a97_bcb5_d62469ce935c.slice/crio-df55e6fb1f5f5c01bee2efc94b017768303ec5d675dae55a448c19b3213a504e WatchSource:0}: Error finding container df55e6fb1f5f5c01bee2efc94b017768303ec5d675dae55a448c19b3213a504e: Status 404 returned error can't find the container with id df55e6fb1f5f5c01bee2efc94b017768303ec5d675dae55a448c19b3213a504e Oct 31 18:36:37.632390 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:37.632356 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" event={"ID":"e7dc18d1-6681-4a97-bcb5-d62469ce935c","Type":"ContainerStarted","Data":"4c5c5ff19f94a0037e6ef6cd0dd6b59a6f45c9b7579b2eced736b844683e922a"} Oct 31 18:36:37.632746 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:37.632417 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" event={"ID":"e7dc18d1-6681-4a97-bcb5-d62469ce935c","Type":"ContainerStarted","Data":"df55e6fb1f5f5c01bee2efc94b017768303ec5d675dae55a448c19b3213a504e"} Oct 31 18:36:37.745908 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:37.745854 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" podStartSLOduration=1.7458377870000001 podStartE2EDuration="1.745837787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:37.745348126 +0000 UTC m=+918.347489270" watchObservedRunningTime="2026-10-31 18:36:37.745837787 +0000 UTC m=+918.347978932" Oct 31 18:36:38.638668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:38.638634 5688 generic.go:356] "Generic (PLEG): container finished" podID="e7dc18d1-6681-4a97-bcb5-d62469ce935c" containerID="4c5c5ff19f94a0037e6ef6cd0dd6b59a6f45c9b7579b2eced736b844683e922a" exitCode=0 Oct 31 18:36:38.639057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:38.638674 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" event={"ID":"e7dc18d1-6681-4a97-bcb5-d62469ce935c","Type":"ContainerDied","Data":"4c5c5ff19f94a0037e6ef6cd0dd6b59a6f45c9b7579b2eced736b844683e922a"} Oct 31 18:36:39.778244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.778212 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:39.892925 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.892899 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r\" (UniqueName: \"kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r\") pod \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " Oct 31 18:36:39.893032 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.892966 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca\") pod \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " Oct 31 18:36:39.893032 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.893011 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca\") pod \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\" (UID: \"e7dc18d1-6681-4a97-bcb5-d62469ce935c\") " Oct 31 18:36:39.893320 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.893293 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca" pod "e7dc18d1-6681-4a97-bcb5-d62469ce935c" (UID: "e7dc18d1-6681-4a97-bcb5-d62469ce935c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:36:39.895059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.895000 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r" pod "e7dc18d1-6681-4a97-bcb5-d62469ce935c" (UID: "e7dc18d1-6681-4a97-bcb5-d62469ce935c"). InnerVolumeSpecName "kube-api-access-fsn8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:36:39.895059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.895048 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca" pod "e7dc18d1-6681-4a97-bcb5-d62469ce935c" (UID: "e7dc18d1-6681-4a97-bcb5-d62469ce935c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:36:39.994538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.994512 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsn8r\" (UniqueName: \"kubernetes.io/projected/e7dc18d1-6681-4a97-bcb5-d62469ce935c-kube-api-access-fsn8r\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:39.994538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.994536 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:39.994700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:39.994545 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dc18d1-6681-4a97-bcb5-d62469ce935c-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:40.649387 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:40.649357 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" Oct 31 18:36:40.649531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:40.649352 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59" event={"ID":"e7dc18d1-6681-4a97-bcb5-d62469ce935c","Type":"ContainerDied","Data":"df55e6fb1f5f5c01bee2efc94b017768303ec5d675dae55a448c19b3213a504e"} Oct 31 18:36:40.649531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:40.649485 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df55e6fb1f5f5c01bee2efc94b017768303ec5d675dae55a448c19b3213a504e" Oct 31 18:36:44.160334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.160299 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw"] Oct 31 18:36:44.160713 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.160700 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e7dc18d1-6681-4a97-bcb5-d62469ce935c" containerName="revoke-keys" Oct 31 18:36:44.160751 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.160713 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dc18d1-6681-4a97-bcb5-d62469ce935c" containerName="revoke-keys" Oct 31 18:36:44.160784 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.160776 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dc18d1-6681-4a97-bcb5-d62469ce935c" containerName="revoke-keys" Oct 31 18:36:44.164144 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.164129 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.218220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.218170 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:36:44.220429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.220397 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw"] Oct 31 18:36:44.267941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.267901 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:36:44.333151 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.333125 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.333322 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.333273 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfph4\" (UniqueName: \"kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.333322 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.333309 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.434449 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.434370 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfph4\" (UniqueName: \"kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.434449 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.434417 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.434641 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.434463 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.435200 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.435163 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.437272 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.437249 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.490930 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.490903 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfph4\" (UniqueName: \"kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4\") pod \"maas-api-revoke-sub-e3b4b4b6-qwhlw\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:44.776536 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:44.776504 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:45.148018 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:45.147905 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw"] Oct 31 18:36:45.150756 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:36:45.150728 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe16175_8a01_4e28_b92c_3f96ae311773.slice/crio-dd726afd4c5e69c19abd4cc65c0bd0eebf479b6147cfa290a4b88db3c6b85bc8 WatchSource:0}: Error finding container dd726afd4c5e69c19abd4cc65c0bd0eebf479b6147cfa290a4b88db3c6b85bc8: Status 404 returned error can't find the container with id dd726afd4c5e69c19abd4cc65c0bd0eebf479b6147cfa290a4b88db3c6b85bc8 Oct 31 18:36:45.674329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:45.674292 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" event={"ID":"fbe16175-8a01-4e28-b92c-3f96ae311773","Type":"ContainerStarted","Data":"2d295beafc936b6fb2dfb2a18d566286f2e7ea70d94218660d3f74c3255fcaeb"} Oct 31 18:36:45.674697 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:45.674356 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" event={"ID":"fbe16175-8a01-4e28-b92c-3f96ae311773","Type":"ContainerStarted","Data":"dd726afd4c5e69c19abd4cc65c0bd0eebf479b6147cfa290a4b88db3c6b85bc8"} Oct 31 18:36:45.788137 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:45.788085 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" podStartSLOduration=1.788069667 podStartE2EDuration="1.788069667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:45.787010867 +0000 UTC m=+926.389152012" watchObservedRunningTime="2026-10-31 18:36:45.788069667 +0000 UTC m=+926.390210814" Oct 31 18:36:46.684213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:46.683313 5688 generic.go:356] "Generic (PLEG): container finished" podID="fbe16175-8a01-4e28-b92c-3f96ae311773" containerID="2d295beafc936b6fb2dfb2a18d566286f2e7ea70d94218660d3f74c3255fcaeb" exitCode=0 Oct 31 18:36:46.684213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:46.683414 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" event={"ID":"fbe16175-8a01-4e28-b92c-3f96ae311773","Type":"ContainerDied","Data":"2d295beafc936b6fb2dfb2a18d566286f2e7ea70d94218660d3f74c3255fcaeb"} Oct 31 18:36:47.864147 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.864107 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:47.891062 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.891033 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca\") pod \"fbe16175-8a01-4e28-b92c-3f96ae311773\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " Oct 31 18:36:47.891201 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.891111 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca\") pod \"fbe16175-8a01-4e28-b92c-3f96ae311773\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " Oct 31 18:36:47.891251 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.891221 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4\" (UniqueName: \"kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4\") pod \"fbe16175-8a01-4e28-b92c-3f96ae311773\" (UID: \"fbe16175-8a01-4e28-b92c-3f96ae311773\") " Oct 31 18:36:47.891759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.891737 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca" pod "fbe16175-8a01-4e28-b92c-3f96ae311773" (UID: "fbe16175-8a01-4e28-b92c-3f96ae311773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:36:47.898223 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.893936 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca" pod "fbe16175-8a01-4e28-b92c-3f96ae311773" (UID: "fbe16175-8a01-4e28-b92c-3f96ae311773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:36:47.898223 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.893972 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4" pod "fbe16175-8a01-4e28-b92c-3f96ae311773" (UID: "fbe16175-8a01-4e28-b92c-3f96ae311773"). InnerVolumeSpecName "kube-api-access-sfph4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:36:47.992902 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.992596 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfph4\" (UniqueName: \"kubernetes.io/projected/fbe16175-8a01-4e28-b92c-3f96ae311773-kube-api-access-sfph4\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:47.992902 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.992631 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:47.992902 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:47.992647 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe16175-8a01-4e28-b92c-3f96ae311773-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:36:48.699648 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:48.699604 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" event={"ID":"fbe16175-8a01-4e28-b92c-3f96ae311773","Type":"ContainerDied","Data":"dd726afd4c5e69c19abd4cc65c0bd0eebf479b6147cfa290a4b88db3c6b85bc8"} Oct 31 18:36:48.699822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:48.699675 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd726afd4c5e69c19abd4cc65c0bd0eebf479b6147cfa290a4b88db3c6b85bc8" Oct 31 18:36:48.699822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:48.699698 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw" Oct 31 18:36:54.904872 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:36:54.904839 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:36:55.990164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:55.990132 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc"] Oct 31 18:36:55.990655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:55.990554 5688 cpu_manager.go:414] "containerMap: removing container" podUID="fbe16175-8a01-4e28-b92c-3f96ae311773" containerName="revoke-keys" Oct 31 18:36:55.990655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:55.990568 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe16175-8a01-4e28-b92c-3f96ae311773" containerName="revoke-keys" Oct 31 18:36:55.990655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:55.990653 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe16175-8a01-4e28-b92c-3f96ae311773" containerName="revoke-keys" Oct 31 18:36:55.994516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:55.994497 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.046374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.046351 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-88085e-openshift-default-dockercfg-96c8d\"" Oct 31 18:36:56.066546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066515 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066563 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066706 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066691 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62hc\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066756 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066724 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066791 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066760 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066791 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066784 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066861 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066812 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066896 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066872 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.066936 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.066923 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.098434 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.098401 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc"] Oct 31 18:36:56.167622 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167596 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62hc\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.167754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167628 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.167754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167651 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.167754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167673 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.167754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167696 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.167754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167739 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167763 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167800 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.167824 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.168064 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.168167 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.168227 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.168377 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.168338 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.169022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.168998 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.170073 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.170046 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.170356 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.170339 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.224427 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.224396 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62hc\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.224644 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.224626 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.309164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.309104 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:56.903067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.903038 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc"] Oct 31 18:36:56.903600 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:36:56.903574 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ffcf50_6bc2_4af4_9aa3_3c77fded0aa2.slice/crio-079cd84837e81c7215105e187308c6a99f138200f4a08e71dc604216c4de0b7c WatchSource:0}: Error finding container 079cd84837e81c7215105e187308c6a99f138200f4a08e71dc604216c4de0b7c: Status 404 returned error can't find the container with id 079cd84837e81c7215105e187308c6a99f138200f4a08e71dc604216c4de0b7c Oct 31 18:36:56.905640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.905605 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:36:56.905744 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.905668 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:36:56.905744 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:56.905694 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:36:57.761269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:57.761227 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" event={"ID":"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2","Type":"ContainerStarted","Data":"12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5"} Oct 31 18:36:57.761801 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:57.761289 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" event={"ID":"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2","Type":"ContainerStarted","Data":"079cd84837e81c7215105e187308c6a99f138200f4a08e71dc604216c4de0b7c"} Oct 31 18:36:57.928764 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:57.928712 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podStartSLOduration=2.928697414 podStartE2EDuration="2.928697414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:57.926559914 +0000 UTC m=+938.528701060" watchObservedRunningTime="2026-10-31 18:36:57.928697414 +0000 UTC m=+938.530838557" Oct 31 18:36:58.309473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:58.309434 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:36:58.311316 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:58.311290 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 18:36:58.311442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:58.311337 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 18:36:59.309807 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:59.309776 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 18:36:59.310233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:36:59.309823 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 18:37:00.310001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:00.309957 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 18:37:00.310484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:00.310026 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 18:37:01.310017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:01.309985 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 18:37:01.310437 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:01.310033 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 18:37:02.309672 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:02.309640 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 18:37:02.309848 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:02.309692 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 18:37:04.352338 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:04.352300 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:37:04.352946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:04.352901 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:37:04.450216 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:04.449503 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:37:04.450635 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:04.450586 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:37:04.457215 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:04.456872 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:37:07.141832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.141800 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl"] Oct 31 18:37:07.145856 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.145832 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.199251 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.199222 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\"" Oct 31 18:37:07.229797 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.229767 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.229955 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.229813 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.229955 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.229933 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.230090 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.230069 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.230165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.230102 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.230165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.230126 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.230165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.230153 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.230360 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.230274 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.230360 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.230351 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwhj\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.250598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.250576 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl"] Oct 31 18:37:07.331589 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331563 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwhj\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331607 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331637 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331653 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331695 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331718 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331818 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331863 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.331971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.331901 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.332173 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.332134 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.332173 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.332156 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.332334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.332245 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.332414 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.332385 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.332544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.332526 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.334535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.334516 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.334845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.334824 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.390806 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.390776 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwhj\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.390897 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.390809 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.459267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.459211 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:07.907417 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.907388 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl"] Oct 31 18:37:07.910707 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:37:07.910681 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5201c7c1_2539_4fdd_a994_bcf5163ea76d.slice/crio-28fb48cc74eb75cb202cf7dbbcc5fd276c9159d7629f98aa1cc8352f56258772 WatchSource:0}: Error finding container 28fb48cc74eb75cb202cf7dbbcc5fd276c9159d7629f98aa1cc8352f56258772: Status 404 returned error can't find the container with id 28fb48cc74eb75cb202cf7dbbcc5fd276c9159d7629f98aa1cc8352f56258772 Oct 31 18:37:07.912885 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.912855 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:37:07.912950 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.912916 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:37:07.912986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:07.912947 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:37:08.860858 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:08.860824 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" event={"ID":"5201c7c1-2539-4fdd-a994-bcf5163ea76d","Type":"ContainerStarted","Data":"8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0"} Oct 31 18:37:08.861389 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:08.860881 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" event={"ID":"5201c7c1-2539-4fdd-a994-bcf5163ea76d","Type":"ContainerStarted","Data":"28fb48cc74eb75cb202cf7dbbcc5fd276c9159d7629f98aa1cc8352f56258772"} Oct 31 18:37:09.029956 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:09.029902 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podStartSLOduration=2.029885123 podStartE2EDuration="2.029885123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:09.029470538 +0000 UTC m=+949.631611695" watchObservedRunningTime="2026-10-31 18:37:09.029885123 +0000 UTC m=+949.632026270" Oct 31 18:37:09.460383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:09.460345 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:09.461728 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:09.461683 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 18:37:09.461863 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:09.461738 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 18:37:10.461105 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:10.460608 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 18:37:10.461105 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:10.460663 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 18:37:11.459793 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:11.459760 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 18:37:11.459975 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:11.459817 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 18:37:12.460083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:12.460047 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 18:37:12.460689 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:12.460275 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 18:37:13.460483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:13.460453 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 18:37:13.460971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:13.460501 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 18:37:14.463408 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:14.463374 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:14.463977 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:14.463742 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:14.464606 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:14.464585 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:37:20.462667 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.462630 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt"] Oct 31 18:37:20.468104 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.468067 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.577714 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.577510 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt"] Oct 31 18:37:20.592122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592094 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592142 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592172 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592243 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzsk\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592297 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592323 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592365 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592421 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.592522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.592483 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.693207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.693162 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.693429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.693413 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.693597 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.693578 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzsk\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.693724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.693710 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.693814 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.693802 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.693922 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.693910 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.694038 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.694026 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.694162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.694149 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.694307 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.694292 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.694778 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.694757 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.695241 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.695005 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.696013 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.695960 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.696376 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.696354 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.696753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.696704 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.699633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.699606 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.699744 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.699720 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.751027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.750980 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzsk\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.751296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.751277 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-962gt\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:20.790400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:20.790366 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:21.244435 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:21.244406 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt"] Oct 31 18:37:21.246141 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:37:21.246116 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eb628e_7cde_44e6_9b9d_21326edc75c9.slice/crio-648ca467ef50dcc7a5a5d77c894624800e45cd0aa0e22a6eca0ea91828ce931a WatchSource:0}: Error finding container 648ca467ef50dcc7a5a5d77c894624800e45cd0aa0e22a6eca0ea91828ce931a: Status 404 returned error can't find the container with id 648ca467ef50dcc7a5a5d77c894624800e45cd0aa0e22a6eca0ea91828ce931a Oct 31 18:37:21.248461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:21.248430 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:37:21.248531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:21.248491 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:37:21.248531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:21.248519 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:37:21.930294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:21.930257 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" event={"ID":"95eb628e-7cde-44e6-9b9d-21326edc75c9","Type":"ContainerStarted","Data":"da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258"} Oct 31 18:37:21.930686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:21.930336 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" event={"ID":"95eb628e-7cde-44e6-9b9d-21326edc75c9","Type":"ContainerStarted","Data":"648ca467ef50dcc7a5a5d77c894624800e45cd0aa0e22a6eca0ea91828ce931a"} Oct 31 18:37:22.055208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:22.052175 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podStartSLOduration=2.0521555989999998 podStartE2EDuration="2.052155599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:22.04973918 +0000 UTC m=+962.651880327" watchObservedRunningTime="2026-10-31 18:37:22.052155599 +0000 UTC m=+962.654296743" Oct 31 18:37:22.791322 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:22.791286 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:22.792921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:22.792894 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 18:37:22.793045 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:22.792945 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 18:37:23.791341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:23.791308 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 18:37:23.791765 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:23.791361 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 18:37:24.790873 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:24.790835 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 18:37:24.791044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:24.790884 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 18:37:24.904051 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:37:24.904006 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:37:25.791500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:25.791468 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 18:37:25.791712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:25.791525 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 18:37:26.791392 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:26.791358 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 18:37:26.791840 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:26.791420 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 18:37:27.791236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:27.791205 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 18:37:27.791484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:27.791253 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 18:37:29.273792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:29.273291 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:29.274316 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:29.274246 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:29.274905 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:29.274887 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:37:48.165326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.165292 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87"] Oct 31 18:37:48.169242 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.169219 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.226525 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.226497 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87"] Oct 31 18:37:48.235072 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.235041 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjpr\" (UniqueName: \"kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.235233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.235211 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.235298 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.235285 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.235343 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.235332 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.235377 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.235353 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.235412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.235399 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.273525 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.273503 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"distinct-w0-a8587d-kserve-self-signed-certs\"" Oct 31 18:37:48.338842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.338789 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.338842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.338845 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.339097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.338882 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.339097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.338924 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjpr\" (UniqueName: \"kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.339097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.339042 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.339304 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.339097 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.339578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.339554 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.339842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.339795 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.340060 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.340041 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.352212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.351050 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.352893 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.352869 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.396695 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.396667 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjpr\" (UniqueName: \"kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr\") pod \"distinct-w0-a8587d-kserve-5b8bc566b6-8rr87\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.481843 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.481769 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:48.895025 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:48.895001 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87"] Oct 31 18:37:48.897864 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:37:48.897837 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c10670a_933d_4f20_a46d_cc4cb63e7091.slice/crio-02c1b1dfdb11c1350494eefad62dd26d2e7566ab44dac70647bd80185b586e2a WatchSource:0}: Error finding container 02c1b1dfdb11c1350494eefad62dd26d2e7566ab44dac70647bd80185b586e2a: Status 404 returned error can't find the container with id 02c1b1dfdb11c1350494eefad62dd26d2e7566ab44dac70647bd80185b586e2a Oct 31 18:37:49.104438 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:49.104402 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" event={"ID":"3c10670a-933d-4f20-a46d-cc4cb63e7091","Type":"ContainerStarted","Data":"59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37"} Oct 31 18:37:49.104659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:49.104447 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" event={"ID":"3c10670a-933d-4f20-a46d-cc4cb63e7091","Type":"ContainerStarted","Data":"02c1b1dfdb11c1350494eefad62dd26d2e7566ab44dac70647bd80185b586e2a"} Oct 31 18:37:49.220968 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:49.220872 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" podStartSLOduration=1.220858566 podStartE2EDuration="1.220858566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:49.218724232 +0000 UTC m=+989.820865368" watchObservedRunningTime="2026-10-31 18:37:49.220858566 +0000 UTC m=+989.822999712" Oct 31 18:37:49.482812 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:49.482784 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:54.906263 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:37:54.906222 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:37:58.481955 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:58.481873 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:58.494323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:58.494299 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:37:59.170609 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:37:59.170579 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 18:38:24.904868 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:38:24.904836 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:38:41.853527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.853494 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287"] Oct 31 18:38:41.858448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.858429 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939061 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939110 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939207 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939253 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939350 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939402 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939439 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939610 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939500 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ms6\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.939610 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.939531 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:41.963005 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:41.962978 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287"] Oct 31 18:38:42.040153 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040123 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040162 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040223 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040252 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040281 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040319 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040353 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040420 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ms6\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040453 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040581 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040573 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040790 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040719 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.040850 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040788 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.041000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.040931 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.041246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.041208 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.042789 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.042753 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.042891 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.042848 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.096586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.096559 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.096810 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.096784 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ms6\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.171490 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.171430 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:42.599156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.599131 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287"] Oct 31 18:38:42.601205 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:38:42.601152 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e77c87_aa2d_40d8_b273_c1547f9b6479.slice/crio-a618dcd382344d7e2c23ad235ffe3ed944138c1ff9fdb57a1469aaeaae4a9683 WatchSource:0}: Error finding container a618dcd382344d7e2c23ad235ffe3ed944138c1ff9fdb57a1469aaeaae4a9683: Status 404 returned error can't find the container with id a618dcd382344d7e2c23ad235ffe3ed944138c1ff9fdb57a1469aaeaae4a9683 Oct 31 18:38:42.603260 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.603223 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:38:42.603349 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.603287 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:38:42.603349 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:42.603313 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:38:43.430857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:43.430809 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" event={"ID":"63e77c87-aa2d-40d8-b273-c1547f9b6479","Type":"ContainerStarted","Data":"6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4"} Oct 31 18:38:43.431359 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:43.430871 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" event={"ID":"63e77c87-aa2d-40d8-b273-c1547f9b6479","Type":"ContainerStarted","Data":"a618dcd382344d7e2c23ad235ffe3ed944138c1ff9fdb57a1469aaeaae4a9683"} Oct 31 18:38:43.652218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:43.652117 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podStartSLOduration=2.6520966440000002 podStartE2EDuration="2.652096644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:43.597394285 +0000 UTC m=+1044.199535427" watchObservedRunningTime="2026-10-31 18:38:43.652096644 +0000 UTC m=+1044.254237789" Oct 31 18:38:44.172461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:44.172434 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:44.173985 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:44.173965 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 18:38:44.174094 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:44.174021 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 18:38:45.172436 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:45.172404 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 18:38:45.172820 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:45.172455 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 18:38:46.172219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:46.172172 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 18:38:46.172450 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:46.172420 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 18:38:47.172390 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:47.172354 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 18:38:47.172566 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:47.172404 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 18:38:48.172006 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:48.171969 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 18:38:48.172283 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:48.172026 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 18:38:49.172915 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:49.172881 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 18:38:49.173320 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:49.172953 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 18:38:50.176793 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:50.176762 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:50.177297 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:50.177056 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:50.177663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:38:50.177637 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:38:54.905116 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:38:54.905080 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:39:09.813851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.813814 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp"] Oct 31 18:39:09.817064 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.817048 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:09.921136 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.921109 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"distinct-w1-da6ea7-kserve-self-signed-certs\"" Oct 31 18:39:09.923153 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.923133 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp"] Oct 31 18:39:09.939798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.939775 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:09.939923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.939823 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:09.939923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.939893 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvbp\" (UniqueName: \"kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:09.940007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.939963 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:09.940057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.940041 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:09.940092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:09.940068 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041274 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041332 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041352 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvbp\" (UniqueName: \"kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041394 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041419 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041443 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041791 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041741 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.041894 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.041846 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.042048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.042026 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.044273 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.044245 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.044582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.044566 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.098550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.098485 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvbp\" (UniqueName: \"kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp\") pod \"distinct-w1-da6ea7-kserve-645f66cf4-266dp\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.128788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.128760 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:10.527393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.527370 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp"] Oct 31 18:39:10.529616 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:39:10.529594 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c93d0a_85c2_4c9d_883b_fa4978bfa528.slice/crio-46476c102c5a1462c120d239e304ce208f5c79948a848a7e7947be15c9cb3c1d WatchSource:0}: Error finding container 46476c102c5a1462c120d239e304ce208f5c79948a848a7e7947be15c9cb3c1d: Status 404 returned error can't find the container with id 46476c102c5a1462c120d239e304ce208f5c79948a848a7e7947be15c9cb3c1d Oct 31 18:39:10.592608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.592577 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" event={"ID":"15c93d0a-85c2-4c9d-883b-fa4978bfa528","Type":"ContainerStarted","Data":"d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f"} Oct 31 18:39:10.592707 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.592623 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" event={"ID":"15c93d0a-85c2-4c9d-883b-fa4978bfa528","Type":"ContainerStarted","Data":"46476c102c5a1462c120d239e304ce208f5c79948a848a7e7947be15c9cb3c1d"} Oct 31 18:39:10.708023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:10.707973 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podStartSLOduration=1.7079603410000002 podStartE2EDuration="1.707960341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:10.705776382 +0000 UTC m=+1071.307917525" watchObservedRunningTime="2026-10-31 18:39:10.707960341 +0000 UTC m=+1071.310101485" Oct 31 18:39:11.129795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:11.129766 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:20.128899 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:20.128863 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:20.141574 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:20.141391 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:20.657903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:20.657875 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:39:23.180529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.180495 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v"] Oct 31 18:39:23.184656 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.184632 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206573 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206609 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206631 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206717 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206743 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206881 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206779 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206881 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206848 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206940 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206894 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.206940 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.206921 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rzj\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.244402 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.244375 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v"] Oct 31 18:39:23.307771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.307741 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.307941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.307787 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.307941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.307824 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.307941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.307850 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.307941 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.307906 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.307968 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308005 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308048 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rzj\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308158 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308283 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308564 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308539 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.308753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308599 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.309009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308819 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.309009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.308883 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.310867 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.310842 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.311030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.310854 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.364420 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.364399 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.364600 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.364581 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rzj\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.497704 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.497681 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:23.926796 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:39:23.926767 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe524cd_93c5_4ad0_b7a6_f5c8f21c0631.slice/crio-af5a0426b14999e2f31bfc47dc2a40c400bd798272c265971cfba9aca3cd7f7b WatchSource:0}: Error finding container af5a0426b14999e2f31bfc47dc2a40c400bd798272c265971cfba9aca3cd7f7b: Status 404 returned error can't find the container with id af5a0426b14999e2f31bfc47dc2a40c400bd798272c265971cfba9aca3cd7f7b Oct 31 18:39:23.929020 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.928987 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:39:23.929106 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.929053 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:39:23.929106 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.929086 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:39:23.947910 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:23.947885 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v"] Oct 31 18:39:24.678630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:24.678588 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" event={"ID":"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631","Type":"ContainerStarted","Data":"ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f"} Oct 31 18:39:24.679083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:24.678642 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" event={"ID":"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631","Type":"ContainerStarted","Data":"af5a0426b14999e2f31bfc47dc2a40c400bd798272c265971cfba9aca3cd7f7b"} Oct 31 18:39:24.904688 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:39:24.904654 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:39:25.498824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:25.498791 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:25.500406 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:25.500387 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 18:39:25.500479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:25.500439 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 18:39:26.498691 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:26.498660 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 18:39:26.499214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:26.498717 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 18:39:27.499023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:27.498988 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 18:39:27.499442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:27.499054 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 18:39:28.498336 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:28.498305 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 18:39:28.498491 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:28.498354 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 18:39:29.498539 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:29.498512 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 18:39:29.498940 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:29.498562 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 18:39:30.498791 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:30.498757 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 18:39:30.499326 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:30.498806 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 18:39:31.502042 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:31.502010 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:31.502455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:31.502385 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:31.503074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:31.503056 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:39:31.667560 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:31.667510 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podStartSLOduration=8.667497394 podStartE2EDuration="8.667497394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:24.897229792 +0000 UTC m=+1085.499370935" watchObservedRunningTime="2026-10-31 18:39:31.667497394 +0000 UTC m=+1092.269638537" Oct 31 18:39:36.710691 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.710653 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp"] Oct 31 18:39:36.714378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.714361 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.769721 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.769687 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:39:36.783324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.783299 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.783458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.783344 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4q6\" (UniqueName: \"kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.783458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.783365 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.819311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.819281 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:39:36.821459 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.821439 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp"] Oct 31 18:39:36.887221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.884462 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4q6\" (UniqueName: \"kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.887221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.884526 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.887221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.884716 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.887221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.885584 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.909231 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.909176 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:36.950306 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:36.950280 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4q6\" (UniqueName: \"kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6\") pod \"maas-api-revoke-sub-a7b7c162-gdnlp\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:37.027338 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:37.027312 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:37.400080 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:37.400054 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp"] Oct 31 18:39:37.401597 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:39:37.401567 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4661d39_dccb_417c_b579_eb984ecf82f9.slice/crio-124dd54030703dc829d14c9c2e0fe0c3055cb901cee777a6b509c94de85d236e WatchSource:0}: Error finding container 124dd54030703dc829d14c9c2e0fe0c3055cb901cee777a6b509c94de85d236e: Status 404 returned error can't find the container with id 124dd54030703dc829d14c9c2e0fe0c3055cb901cee777a6b509c94de85d236e Oct 31 18:39:37.746003 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:37.745974 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" event={"ID":"a4661d39-dccb-417c-b579-eb984ecf82f9","Type":"ContainerStarted","Data":"fde0af346780f83afc4b992230c345f2936d1d852d7c0de4eb9a8668286274d1"} Oct 31 18:39:37.746403 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:37.746024 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" event={"ID":"a4661d39-dccb-417c-b579-eb984ecf82f9","Type":"ContainerStarted","Data":"124dd54030703dc829d14c9c2e0fe0c3055cb901cee777a6b509c94de85d236e"} Oct 31 18:39:37.874206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:37.873226 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" podStartSLOduration=1.8732027900000001 podStartE2EDuration="1.87320279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:37.869410983 +0000 UTC m=+1098.471552129" watchObservedRunningTime="2026-10-31 18:39:37.87320279 +0000 UTC m=+1098.475343929" Oct 31 18:39:38.754522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:38.754488 5688 generic.go:356] "Generic (PLEG): container finished" podID="a4661d39-dccb-417c-b579-eb984ecf82f9" containerID="fde0af346780f83afc4b992230c345f2936d1d852d7c0de4eb9a8668286274d1" exitCode=0 Oct 31 18:39:38.754988 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:38.754541 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" event={"ID":"a4661d39-dccb-417c-b579-eb984ecf82f9","Type":"ContainerDied","Data":"fde0af346780f83afc4b992230c345f2936d1d852d7c0de4eb9a8668286274d1"} Oct 31 18:39:39.904810 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:39.904781 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:40.030174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.030104 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca\") pod \"a4661d39-dccb-417c-b579-eb984ecf82f9\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " Oct 31 18:39:40.030174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.030174 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca\") pod \"a4661d39-dccb-417c-b579-eb984ecf82f9\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " Oct 31 18:39:40.030407 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.030265 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6\" (UniqueName: \"kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6\") pod \"a4661d39-dccb-417c-b579-eb984ecf82f9\" (UID: \"a4661d39-dccb-417c-b579-eb984ecf82f9\") " Oct 31 18:39:40.030485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.030466 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca" pod "a4661d39-dccb-417c-b579-eb984ecf82f9" (UID: "a4661d39-dccb-417c-b579-eb984ecf82f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:39:40.030782 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.030764 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:39:40.033345 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.033321 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6" pod "a4661d39-dccb-417c-b579-eb984ecf82f9" (UID: "a4661d39-dccb-417c-b579-eb984ecf82f9"). InnerVolumeSpecName "kube-api-access-cc4q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:39:40.039369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.039346 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca" pod "a4661d39-dccb-417c-b579-eb984ecf82f9" (UID: "a4661d39-dccb-417c-b579-eb984ecf82f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:39:40.132072 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.132047 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4661d39-dccb-417c-b579-eb984ecf82f9-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:39:40.132072 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.132077 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc4q6\" (UniqueName: \"kubernetes.io/projected/a4661d39-dccb-417c-b579-eb984ecf82f9-kube-api-access-cc4q6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:39:40.765562 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.765530 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" event={"ID":"a4661d39-dccb-417c-b579-eb984ecf82f9","Type":"ContainerDied","Data":"124dd54030703dc829d14c9c2e0fe0c3055cb901cee777a6b509c94de85d236e"} Oct 31 18:39:40.765753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.765588 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="124dd54030703dc829d14c9c2e0fe0c3055cb901cee777a6b509c94de85d236e" Oct 31 18:39:40.765753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:40.765539 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp" Oct 31 18:39:50.351140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.351098 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn"] Oct 31 18:39:50.351744 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.351723 5688 cpu_manager.go:414] "containerMap: removing container" podUID="a4661d39-dccb-417c-b579-eb984ecf82f9" containerName="revoke-keys" Oct 31 18:39:50.351824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.351747 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4661d39-dccb-417c-b579-eb984ecf82f9" containerName="revoke-keys" Oct 31 18:39:50.351920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.351907 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4661d39-dccb-417c-b579-eb984ecf82f9" containerName="revoke-keys" Oct 31 18:39:50.355268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.355247 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.447378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.447323 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.447378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.447366 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vg92\" (UniqueName: \"kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.447637 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.447577 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.459038 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.459015 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:39:50.459157 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.459142 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:39:50.461288 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.461270 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn"] Oct 31 18:39:50.549067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.549039 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.549067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.549068 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vg92\" (UniqueName: \"kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.549290 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.549229 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.549837 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.549818 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.551627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.551601 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.606208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.606129 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vg92\" (UniqueName: \"kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92\") pod \"maas-api-revoke-sub-9f0f4379-q2rwn\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:50.667472 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:50.667448 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:51.055708 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:51.050238 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn"] Oct 31 18:39:51.827339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:51.827298 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" event={"ID":"c1340650-82c6-45f7-9b08-7578575eab13","Type":"ContainerStarted","Data":"6c8195197c56eac0b177c82b210acbff68001b7942c1d319f7f18c58b1f17d72"} Oct 31 18:39:51.827838 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:51.827355 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" event={"ID":"c1340650-82c6-45f7-9b08-7578575eab13","Type":"ContainerStarted","Data":"f676f9ed3a5c76e94097a79de8a5544813bc5a199ef447c867d4a27c4afa5603"} Oct 31 18:39:51.941625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:51.941570 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" podStartSLOduration=1.9415535149999998 podStartE2EDuration="1.941553515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:51.93972092 +0000 UTC m=+1112.541862066" watchObservedRunningTime="2026-10-31 18:39:51.941553515 +0000 UTC m=+1112.543694663" Oct 31 18:39:52.834150 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:52.834118 5688 generic.go:356] "Generic (PLEG): container finished" podID="c1340650-82c6-45f7-9b08-7578575eab13" containerID="6c8195197c56eac0b177c82b210acbff68001b7942c1d319f7f18c58b1f17d72" exitCode=0 Oct 31 18:39:52.834557 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:52.834214 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" event={"ID":"c1340650-82c6-45f7-9b08-7578575eab13","Type":"ContainerDied","Data":"6c8195197c56eac0b177c82b210acbff68001b7942c1d319f7f18c58b1f17d72"} Oct 31 18:39:53.971790 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:53.971762 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:54.085861 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.085832 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca\") pod \"c1340650-82c6-45f7-9b08-7578575eab13\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " Oct 31 18:39:54.086009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.085884 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca\") pod \"c1340650-82c6-45f7-9b08-7578575eab13\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " Oct 31 18:39:54.086009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.085928 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92\" (UniqueName: \"kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92\") pod \"c1340650-82c6-45f7-9b08-7578575eab13\" (UID: \"c1340650-82c6-45f7-9b08-7578575eab13\") " Oct 31 18:39:54.086344 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.086313 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca" pod "c1340650-82c6-45f7-9b08-7578575eab13" (UID: "c1340650-82c6-45f7-9b08-7578575eab13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:39:54.093098 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.093035 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca" pod "c1340650-82c6-45f7-9b08-7578575eab13" (UID: "c1340650-82c6-45f7-9b08-7578575eab13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:39:54.093250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.093150 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92" pod "c1340650-82c6-45f7-9b08-7578575eab13" (UID: "c1340650-82c6-45f7-9b08-7578575eab13"). InnerVolumeSpecName "kube-api-access-5vg92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:39:54.187805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.187771 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:39:54.187942 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.187822 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1340650-82c6-45f7-9b08-7578575eab13-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:39:54.187942 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.187838 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vg92\" (UniqueName: \"kubernetes.io/projected/c1340650-82c6-45f7-9b08-7578575eab13-kube-api-access-5vg92\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:39:54.846244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.846210 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" Oct 31 18:39:54.846429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.846178 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn" event={"ID":"c1340650-82c6-45f7-9b08-7578575eab13","Type":"ContainerDied","Data":"f676f9ed3a5c76e94097a79de8a5544813bc5a199ef447c867d4a27c4afa5603"} Oct 31 18:39:54.846429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:39:54.846338 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f676f9ed3a5c76e94097a79de8a5544813bc5a199ef447c867d4a27c4afa5603" Oct 31 18:39:54.904983 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:39:54.904950 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:40:06.544094 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.544062 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk"] Oct 31 18:40:06.544585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.544568 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c1340650-82c6-45f7-9b08-7578575eab13" containerName="revoke-keys" Oct 31 18:40:06.544623 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.544586 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1340650-82c6-45f7-9b08-7578575eab13" containerName="revoke-keys" Oct 31 18:40:06.544674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.544666 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1340650-82c6-45f7-9b08-7578575eab13" containerName="revoke-keys" Oct 31 18:40:06.547619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.547600 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.605029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.604984 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk"] Oct 31 18:40:06.651421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.651392 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:40:06.651550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.651503 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:40:06.661397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.661362 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.661627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.661599 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.661742 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.661653 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6p7\" (UniqueName: \"kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.762753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.762725 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.762861 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.762764 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6p7\" (UniqueName: \"kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.762861 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.762845 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.763544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.763522 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.765129 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.765111 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.822230 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.822152 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6p7\" (UniqueName: \"kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7\") pod \"maas-api-revoke-sub-89c837a9-qknhk\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:06.861053 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:06.861030 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:07.233368 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:07.233343 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk"] Oct 31 18:40:07.235159 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:40:07.235135 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7e1faa_342d_4954_8394_c4b33038f3eb.slice/crio-ef3be1149ab00c36b1a1da376f85f4ea3aae3c7ef8e5accf38707b469156ccfd WatchSource:0}: Error finding container ef3be1149ab00c36b1a1da376f85f4ea3aae3c7ef8e5accf38707b469156ccfd: Status 404 returned error can't find the container with id ef3be1149ab00c36b1a1da376f85f4ea3aae3c7ef8e5accf38707b469156ccfd Oct 31 18:40:07.954119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:07.954043 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" event={"ID":"5f7e1faa-342d-4954-8394-c4b33038f3eb","Type":"ContainerStarted","Data":"ce0b7bc044cd4923d063f2ec33736d6a02c50d82cec8927ace56f3b5eda9ea40"} Oct 31 18:40:07.954119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:07.954104 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" event={"ID":"5f7e1faa-342d-4954-8394-c4b33038f3eb","Type":"ContainerStarted","Data":"ef3be1149ab00c36b1a1da376f85f4ea3aae3c7ef8e5accf38707b469156ccfd"} Oct 31 18:40:08.069042 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:08.068992 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" podStartSLOduration=2.06897888 podStartE2EDuration="2.06897888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:08.066422054 +0000 UTC m=+1128.668563199" watchObservedRunningTime="2026-10-31 18:40:08.06897888 +0000 UTC m=+1128.671120014" Oct 31 18:40:08.960730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:08.960696 5688 generic.go:356] "Generic (PLEG): container finished" podID="5f7e1faa-342d-4954-8394-c4b33038f3eb" containerID="ce0b7bc044cd4923d063f2ec33736d6a02c50d82cec8927ace56f3b5eda9ea40" exitCode=0 Oct 31 18:40:08.961088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:08.960769 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" event={"ID":"5f7e1faa-342d-4954-8394-c4b33038f3eb","Type":"ContainerDied","Data":"ce0b7bc044cd4923d063f2ec33736d6a02c50d82cec8927ace56f3b5eda9ea40"} Oct 31 18:40:10.113945 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.113915 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:10.199343 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.199317 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca\") pod \"5f7e1faa-342d-4954-8394-c4b33038f3eb\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " Oct 31 18:40:10.199478 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.199404 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7\" (UniqueName: \"kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7\") pod \"5f7e1faa-342d-4954-8394-c4b33038f3eb\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " Oct 31 18:40:10.199541 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.199511 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca\") pod \"5f7e1faa-342d-4954-8394-c4b33038f3eb\" (UID: \"5f7e1faa-342d-4954-8394-c4b33038f3eb\") " Oct 31 18:40:10.199719 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.199681 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca" pod "5f7e1faa-342d-4954-8394-c4b33038f3eb" (UID: "5f7e1faa-342d-4954-8394-c4b33038f3eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:40:10.199813 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.199758 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:10.201493 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.201470 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca" pod "5f7e1faa-342d-4954-8394-c4b33038f3eb" (UID: "5f7e1faa-342d-4954-8394-c4b33038f3eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:40:10.201624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.201605 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7" pod "5f7e1faa-342d-4954-8394-c4b33038f3eb" (UID: "5f7e1faa-342d-4954-8394-c4b33038f3eb"). InnerVolumeSpecName "kube-api-access-7b6p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:40:10.300547 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.300519 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e1faa-342d-4954-8394-c4b33038f3eb-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:10.300547 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.300548 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6p7\" (UniqueName: \"kubernetes.io/projected/5f7e1faa-342d-4954-8394-c4b33038f3eb-kube-api-access-7b6p7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:10.974385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.974359 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" Oct 31 18:40:10.974542 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.974405 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk" event={"ID":"5f7e1faa-342d-4954-8394-c4b33038f3eb","Type":"ContainerDied","Data":"ef3be1149ab00c36b1a1da376f85f4ea3aae3c7ef8e5accf38707b469156ccfd"} Oct 31 18:40:10.974542 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:10.974451 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3be1149ab00c36b1a1da376f85f4ea3aae3c7ef8e5accf38707b469156ccfd" Oct 31 18:40:20.642082 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.642050 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq"] Oct 31 18:40:20.642492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.642445 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5f7e1faa-342d-4954-8394-c4b33038f3eb" containerName="revoke-keys" Oct 31 18:40:20.642492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.642457 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7e1faa-342d-4954-8394-c4b33038f3eb" containerName="revoke-keys" Oct 31 18:40:20.642561 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.642553 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7e1faa-342d-4954-8394-c4b33038f3eb" containerName="revoke-keys" Oct 31 18:40:20.645985 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.645965 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.698028 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.698007 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a8587d-openshift-default-dockercfg-zdp6b\"" Oct 31 18:40:20.725895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.725863 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.725915 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726020 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdrbz\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726053 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726070 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726123 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726220 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726241 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.726317 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.726277 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.750798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.750773 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq"] Oct 31 18:40:20.827580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827550 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdrbz\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827587 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827604 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827622 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827680 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827700 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827977 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827749 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827977 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827817 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.827977 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.827846 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.828140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.828008 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.828140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.828028 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.828295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.828264 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.828333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.828317 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.828481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.828462 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.830126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.830104 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.830338 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.830322 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.883531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.883508 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.883693 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.883674 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdrbz\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:20.959311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:20.959258 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:21.388178 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:21.388154 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq"] Oct 31 18:40:21.390339 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:40:21.390317 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b48bcae_d1cd_4ab6_87c0_fae1080e6227.slice/crio-acfca5fa1d69bcf8e483ac5210bee6ed14dfe05b526db8c2b2a739a9bcf90e94 WatchSource:0}: Error finding container acfca5fa1d69bcf8e483ac5210bee6ed14dfe05b526db8c2b2a739a9bcf90e94: Status 404 returned error can't find the container with id acfca5fa1d69bcf8e483ac5210bee6ed14dfe05b526db8c2b2a739a9bcf90e94 Oct 31 18:40:21.392614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:21.392574 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:40:21.392716 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:21.392660 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:40:21.392764 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:21.392731 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:40:22.063855 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.063813 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" event={"ID":"3b48bcae-d1cd-4ab6-87c0-fae1080e6227","Type":"ContainerStarted","Data":"47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3"} Oct 31 18:40:22.064301 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.063875 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" event={"ID":"3b48bcae-d1cd-4ab6-87c0-fae1080e6227","Type":"ContainerStarted","Data":"acfca5fa1d69bcf8e483ac5210bee6ed14dfe05b526db8c2b2a739a9bcf90e94"} Oct 31 18:40:22.181566 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.181498 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podStartSLOduration=2.181483387 podStartE2EDuration="2.181483387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:22.17857266 +0000 UTC m=+1142.780713801" watchObservedRunningTime="2026-10-31 18:40:22.181483387 +0000 UTC m=+1142.783624530" Oct 31 18:40:22.527682 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.527650 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6"] Oct 31 18:40:22.531065 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.531047 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.561801 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.561724 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rt7n\" (UniqueName: \"kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.562095 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.562068 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.562246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.562117 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.585387 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.585340 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:40:22.585387 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.585368 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:40:22.589123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.589090 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6"] Oct 31 18:40:22.664042 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.664006 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rt7n\" (UniqueName: \"kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.664275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.664211 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.664356 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.664276 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.665083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.665056 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.666834 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.666810 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.720832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.720807 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rt7n\" (UniqueName: \"kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n\") pod \"maas-api-revoke-sub-888b8e32-hvnf6\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.868227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.868138 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:22.969233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.967633 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:22.971503 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.971478 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 18:40:22.971619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:22.971526 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 18:40:23.247246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.246361 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6"] Oct 31 18:40:23.673174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.673143 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp"] Oct 31 18:40:23.676991 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.676974 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.767018 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.766987 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp"] Oct 31 18:40:23.775537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775498 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775691 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775553 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775691 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775601 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775691 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775657 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775691 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775689 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775917 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775734 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775917 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775815 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775917 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775860 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.775917 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.775884 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876539 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876502 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876556 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876603 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876685 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876728 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876755 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876789 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876816 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.876880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.876823 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.877149 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.877125 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.877266 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.877251 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.877349 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.877322 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.877483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.877378 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.877483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.877448 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.879450 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.879432 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.879541 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.879495 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.960255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.956273 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.961609 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.961017 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 18:40:23.961609 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.961066 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 18:40:23.962870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.962845 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:23.990821 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:23.990798 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:24.122206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.118227 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" event={"ID":"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3","Type":"ContainerStarted","Data":"53f436ad491608a6390b1761c0773dc7fcf3ceb7aec6af46c9dc5ae426a8abf0"} Oct 31 18:40:24.122206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.118286 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" event={"ID":"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3","Type":"ContainerStarted","Data":"b966111516022d6a5039eb5f8aca255fff30a83d57717f281ff9e87d4bca3eaa"} Oct 31 18:40:24.542390 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.540011 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" podStartSLOduration=2.539990802 podStartE2EDuration="2.539990802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:24.264684812 +0000 UTC m=+1144.866825956" watchObservedRunningTime="2026-10-31 18:40:24.539990802 +0000 UTC m=+1145.142131946" Oct 31 18:40:24.552262 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.548641 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp"] Oct 31 18:40:24.572213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.569027 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:40:24.572213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.569123 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:40:24.572213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.569167 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:40:24.906325 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:40:24.906288 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:40:24.962596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.962560 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 18:40:24.962760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:24.962628 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 18:40:25.123331 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.123297 5688 generic.go:356] "Generic (PLEG): container finished" podID="b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" containerID="53f436ad491608a6390b1761c0773dc7fcf3ceb7aec6af46c9dc5ae426a8abf0" exitCode=0 Oct 31 18:40:25.123512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.123376 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" event={"ID":"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3","Type":"ContainerDied","Data":"53f436ad491608a6390b1761c0773dc7fcf3ceb7aec6af46c9dc5ae426a8abf0"} Oct 31 18:40:25.125061 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.125037 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" event={"ID":"89185377-5b72-49b9-a9f9-97911339f08f","Type":"ContainerStarted","Data":"5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38"} Oct 31 18:40:25.125173 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.125081 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" event={"ID":"89185377-5b72-49b9-a9f9-97911339f08f","Type":"ContainerStarted","Data":"fea1b39b1a479ea6c3dbfd527f5fd483d4bdad2cb88253d1a3ce2260b32e2659"} Oct 31 18:40:25.386214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.382092 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podStartSLOduration=2.382072114 podStartE2EDuration="2.382072114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:25.365652606 +0000 UTC m=+1145.967793750" watchObservedRunningTime="2026-10-31 18:40:25.382072114 +0000 UTC m=+1145.984213273" Oct 31 18:40:25.982226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.981622 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 18:40:25.982226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.981683 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 18:40:25.992148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.991689 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:25.992984 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.992962 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 18:40:25.993100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:25.992999 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 18:40:26.416794 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.416769 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:26.530929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.530848 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n\" (UniqueName: \"kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n\") pod \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " Oct 31 18:40:26.531089 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.531013 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca\") pod \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " Oct 31 18:40:26.531089 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.531048 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca\") pod \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\" (UID: \"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3\") " Oct 31 18:40:26.531475 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.531449 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca" pod "b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" (UID: "b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:40:26.546205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.537594 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n" pod "b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" (UID: "b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3"). InnerVolumeSpecName "kube-api-access-5rt7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:40:26.546205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.542573 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca" pod "b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" (UID: "b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:40:26.632112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.632084 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:26.632112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.632110 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:26.632407 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.632121 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rt7n\" (UniqueName: \"kubernetes.io/projected/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3-kube-api-access-5rt7n\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:26.969264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.968340 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 18:40:26.969264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.968397 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 18:40:26.992426 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.992390 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 18:40:26.992800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:26.992444 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 18:40:27.167220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.148949 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" Oct 31 18:40:27.167220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.160240 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6" event={"ID":"b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3","Type":"ContainerDied","Data":"b966111516022d6a5039eb5f8aca255fff30a83d57717f281ff9e87d4bca3eaa"} Oct 31 18:40:27.167220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.160309 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b966111516022d6a5039eb5f8aca255fff30a83d57717f281ff9e87d4bca3eaa" Oct 31 18:40:27.960569 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.960532 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 18:40:27.960781 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.960590 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 18:40:27.991759 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.991731 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 18:40:27.991896 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:27.991784 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 18:40:28.976636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:28.976590 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:28.978775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:28.978752 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:28.980882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:28.980862 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:40:28.995740 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:28.995705 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 18:40:28.995903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:28.995761 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 18:40:29.991656 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:29.991622 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 18:40:29.992263 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:29.992221 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 18:40:31.001209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:30.993577 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 18:40:31.001209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:30.993637 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 18:40:32.254139 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:32.254106 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:32.254714 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:32.254691 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:32.255471 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:32.255450 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:40:33.270500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.270432 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn"] Oct 31 18:40:33.271221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.271175 5688 cpu_manager.go:414] "containerMap: removing container" podUID="b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" containerName="revoke-keys" Oct 31 18:40:33.271221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.271220 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" containerName="revoke-keys" Oct 31 18:40:33.271426 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.271348 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" containerName="revoke-keys" Oct 31 18:40:33.274860 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.274836 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.330563 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.330526 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn"] Oct 31 18:40:33.377748 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.377708 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:40:33.377947 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.377781 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:40:33.414338 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.414298 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.414521 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.414498 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.414575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.414542 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbprn\" (UniqueName: \"kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.515282 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.515251 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.515430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.515355 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.515430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.515385 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbprn\" (UniqueName: \"kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.515912 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.515893 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.517923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.517894 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.572458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.572391 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbprn\" (UniqueName: \"kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn\") pod \"maas-api-revoke-sub-b5d6538f-z4lgn\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.588118 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.588099 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:33.961112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:33.961087 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn"] Oct 31 18:40:33.962595 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:40:33.962571 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0721eb41_5043_4c2b_9e03_89339507c4c4.slice/crio-1aa572d567a4e21866b234ec887a8d1a68d75823cbea3c351f14e868dfed9ee1 WatchSource:0}: Error finding container 1aa572d567a4e21866b234ec887a8d1a68d75823cbea3c351f14e868dfed9ee1: Status 404 returned error can't find the container with id 1aa572d567a4e21866b234ec887a8d1a68d75823cbea3c351f14e868dfed9ee1 Oct 31 18:40:34.218546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:34.218470 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" event={"ID":"0721eb41-5043-4c2b-9e03-89339507c4c4","Type":"ContainerStarted","Data":"1aa572d567a4e21866b234ec887a8d1a68d75823cbea3c351f14e868dfed9ee1"} Oct 31 18:40:35.226223 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:35.226130 5688 generic.go:356] "Generic (PLEG): container finished" podID="0721eb41-5043-4c2b-9e03-89339507c4c4" containerID="099131dc6b4bd3d36f14004bdd7724f8b927c0f0d77b8faa18948e880dcacb4a" exitCode=0 Oct 31 18:40:35.226570 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:35.226232 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" event={"ID":"0721eb41-5043-4c2b-9e03-89339507c4c4","Type":"ContainerDied","Data":"099131dc6b4bd3d36f14004bdd7724f8b927c0f0d77b8faa18948e880dcacb4a"} Oct 31 18:40:36.379878 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.379858 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:36.550749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.550678 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn\" (UniqueName: \"kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn\") pod \"0721eb41-5043-4c2b-9e03-89339507c4c4\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " Oct 31 18:40:36.550749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.550723 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca\") pod \"0721eb41-5043-4c2b-9e03-89339507c4c4\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " Oct 31 18:40:36.550965 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.550759 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca\") pod \"0721eb41-5043-4c2b-9e03-89339507c4c4\" (UID: \"0721eb41-5043-4c2b-9e03-89339507c4c4\") " Oct 31 18:40:36.551156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.551131 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca" pod "0721eb41-5043-4c2b-9e03-89339507c4c4" (UID: "0721eb41-5043-4c2b-9e03-89339507c4c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:40:36.552914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.552890 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca" pod "0721eb41-5043-4c2b-9e03-89339507c4c4" (UID: "0721eb41-5043-4c2b-9e03-89339507c4c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:40:36.552914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.552898 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn" pod "0721eb41-5043-4c2b-9e03-89339507c4c4" (UID: "0721eb41-5043-4c2b-9e03-89339507c4c4"). InnerVolumeSpecName "kube-api-access-wbprn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:40:36.651888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.651864 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:36.651888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.651885 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbprn\" (UniqueName: \"kubernetes.io/projected/0721eb41-5043-4c2b-9e03-89339507c4c4-kube-api-access-wbprn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:36.652048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:36.651894 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0721eb41-5043-4c2b-9e03-89339507c4c4-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:37.239154 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:37.239129 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" Oct 31 18:40:37.239346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:37.239167 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn" event={"ID":"0721eb41-5043-4c2b-9e03-89339507c4c4","Type":"ContainerDied","Data":"1aa572d567a4e21866b234ec887a8d1a68d75823cbea3c351f14e868dfed9ee1"} Oct 31 18:40:37.239346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:37.239229 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa572d567a4e21866b234ec887a8d1a68d75823cbea3c351f14e868dfed9ee1" Oct 31 18:40:40.060624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:40.060593 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q"] Oct 31 18:40:40.110513 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:40.110485 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e12ee39-7cc5q"] Oct 31 18:40:41.565441 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:41.565405 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f3d2d0-4bd2-4420-8dac-294ba3caebd2" path="/var/lib/kubelet/pods/04f3d2d0-4bd2-4420-8dac-294ba3caebd2/volumes" Oct 31 18:40:51.357803 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.357765 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn"] Oct 31 18:40:51.358421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.358402 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0721eb41-5043-4c2b-9e03-89339507c4c4" containerName="revoke-keys" Oct 31 18:40:51.358515 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.358422 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0721eb41-5043-4c2b-9e03-89339507c4c4" containerName="revoke-keys" Oct 31 18:40:51.358584 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.358571 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0721eb41-5043-4c2b-9e03-89339507c4c4" containerName="revoke-keys" Oct 31 18:40:51.362882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.362857 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.408546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.408518 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pks7s\" (UniqueName: \"kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.408741 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.408715 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.408885 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.408868 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.418796 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.418772 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn"] Oct 31 18:40:51.464594 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.464571 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:40:51.464755 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.464740 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:40:51.509732 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.509706 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.509828 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.509805 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.509885 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.509849 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pks7s\" (UniqueName: \"kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.510448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.510386 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.512062 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.512035 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.566396 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.566375 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pks7s\" (UniqueName: \"kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s\") pod \"maas-api-revoke-sub-6badabd3-5fbtn\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:51.675608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:51.675528 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:52.105384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:52.104796 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn"] Oct 31 18:40:52.348414 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:52.348326 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" event={"ID":"254e50e9-df69-4ff7-8846-5ad6822f7723","Type":"ContainerStarted","Data":"243bd1ad03a55a382252918262e344aba5b9f70f096417d6abed842d9af20721"} Oct 31 18:40:53.358671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:53.358633 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" event={"ID":"254e50e9-df69-4ff7-8846-5ad6822f7723","Type":"ContainerStarted","Data":"8de1c788fe6ed8489642fa39ad8ed033a4d8b495c7eaa2577c02bf4317ac90a7"} Oct 31 18:40:54.049210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:54.048566 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" podStartSLOduration=3.048546072 podStartE2EDuration="3.048546072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:53.493424745 +0000 UTC m=+1174.095565891" watchObservedRunningTime="2026-10-31 18:40:54.048546072 +0000 UTC m=+1174.650687217" Oct 31 18:40:54.058208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:54.051043 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt"] Oct 31 18:40:54.109717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:54.109655 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7443e1-c78xt"] Oct 31 18:40:54.237388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:54.234327 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 18:40:54.386837 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:54.386751 5688 generic.go:356] "Generic (PLEG): container finished" podID="254e50e9-df69-4ff7-8846-5ad6822f7723" containerID="8de1c788fe6ed8489642fa39ad8ed033a4d8b495c7eaa2577c02bf4317ac90a7" exitCode=0 Oct 31 18:40:54.387375 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:54.387355 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" event={"ID":"254e50e9-df69-4ff7-8846-5ad6822f7723","Type":"ContainerDied","Data":"8de1c788fe6ed8489642fa39ad8ed033a4d8b495c7eaa2577c02bf4317ac90a7"} Oct 31 18:40:54.904386 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:40:54.904350 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:40:55.523621 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.523599 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:55.565261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.565221 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5452e03a-8fcf-4f18-ba50-783fad74bc0a" path="/var/lib/kubelet/pods/5452e03a-8fcf-4f18-ba50-783fad74bc0a/volumes" Oct 31 18:40:55.636842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.636817 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca\") pod \"254e50e9-df69-4ff7-8846-5ad6822f7723\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " Oct 31 18:40:55.637029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.637005 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s\" (UniqueName: \"kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s\") pod \"254e50e9-df69-4ff7-8846-5ad6822f7723\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " Oct 31 18:40:55.637255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.637235 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca\") pod \"254e50e9-df69-4ff7-8846-5ad6822f7723\" (UID: \"254e50e9-df69-4ff7-8846-5ad6822f7723\") " Oct 31 18:40:55.637758 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.637663 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca" pod "254e50e9-df69-4ff7-8846-5ad6822f7723" (UID: "254e50e9-df69-4ff7-8846-5ad6822f7723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:40:55.638303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.638282 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:55.639708 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.639685 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s" pod "254e50e9-df69-4ff7-8846-5ad6822f7723" (UID: "254e50e9-df69-4ff7-8846-5ad6822f7723"). InnerVolumeSpecName "kube-api-access-pks7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:40:55.640247 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.640217 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca" pod "254e50e9-df69-4ff7-8846-5ad6822f7723" (UID: "254e50e9-df69-4ff7-8846-5ad6822f7723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:40:55.738938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.738909 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/254e50e9-df69-4ff7-8846-5ad6822f7723-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:55.738938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:55.738939 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pks7s\" (UniqueName: \"kubernetes.io/projected/254e50e9-df69-4ff7-8846-5ad6822f7723-kube-api-access-pks7s\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:40:56.405839 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:56.405801 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" Oct 31 18:40:56.405839 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:56.405821 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn" event={"ID":"254e50e9-df69-4ff7-8846-5ad6822f7723","Type":"ContainerDied","Data":"243bd1ad03a55a382252918262e344aba5b9f70f096417d6abed842d9af20721"} Oct 31 18:40:56.406035 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:40:56.405889 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243bd1ad03a55a382252918262e344aba5b9f70f096417d6abed842d9af20721" Oct 31 18:41:02.077693 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.077615 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72"] Oct 31 18:41:02.078124 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.078109 5688 cpu_manager.go:414] "containerMap: removing container" podUID="254e50e9-df69-4ff7-8846-5ad6822f7723" containerName="revoke-keys" Oct 31 18:41:02.078172 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.078126 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254e50e9-df69-4ff7-8846-5ad6822f7723" containerName="revoke-keys" Oct 31 18:41:02.078248 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.078236 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="254e50e9-df69-4ff7-8846-5ad6822f7723" containerName="revoke-keys" Oct 31 18:41:02.081333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.081318 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.134752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.134692 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:41:02.137976 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.137951 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72"] Oct 31 18:41:02.184373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.184341 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:41:02.205873 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.205850 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7dj\" (UniqueName: \"kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.206002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.205880 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.206002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.205971 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.306787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.306759 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7dj\" (UniqueName: \"kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.306787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.306791 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.306959 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.306853 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.307504 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.307486 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.309256 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.309240 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.363078 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.363022 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7dj\" (UniqueName: \"kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj\") pod \"maas-api-revoke-sub-b4a1ae12-k4t72\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.393409 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.393388 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:02.773970 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:02.773946 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72"] Oct 31 18:41:02.775872 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:02.775839 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294ca253_4a8a_4f07_8081_5dc30166e590.slice/crio-2215daf562c0c69c814d39c00ed36221e5b32e7c7091d5a3544db27d9d72a71b WatchSource:0}: Error finding container 2215daf562c0c69c814d39c00ed36221e5b32e7c7091d5a3544db27d9d72a71b: Status 404 returned error can't find the container with id 2215daf562c0c69c814d39c00ed36221e5b32e7c7091d5a3544db27d9d72a71b Oct 31 18:41:03.286521 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.286489 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8"] Oct 31 18:41:03.290175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.290125 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.295141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.295122 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd"] Oct 31 18:41:03.298750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.298625 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.342040 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.342013 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-da6ea7-openshift-default-dockercfg-z7k66\"" Oct 31 18:41:03.374466 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.374440 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8"] Oct 31 18:41:03.426331 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426303 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hc9h\" (UniqueName: \"kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.426696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426369 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426401 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.426696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426479 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426504 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426617 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426661 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426997 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426737 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426997 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426773 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.426997 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426790 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426997 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426827 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4nk\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.426997 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.426956 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.435174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.435151 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd"] Oct 31 18:41:03.455031 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.455004 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" event={"ID":"294ca253-4a8a-4f07-8081-5dc30166e590","Type":"ContainerStarted","Data":"40c1a43a4f8cfbd93c61aea93c8c522151f84058dd81cc5d6f5972d86c4bb6eb"} Oct 31 18:41:03.455152 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.455056 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" event={"ID":"294ca253-4a8a-4f07-8081-5dc30166e590","Type":"ContainerStarted","Data":"2215daf562c0c69c814d39c00ed36221e5b32e7c7091d5a3544db27d9d72a71b"} Oct 31 18:41:03.528168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528140 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hc9h\" (UniqueName: \"kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.528381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528178 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528242 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.528381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528279 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528300 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528359 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528394 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528446 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528500 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.528631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528534 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528601 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4nk\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528696 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528721 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.528976 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.528884 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.529115 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.529088 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.529229 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.529140 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.529229 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.529168 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.529428 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.529365 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.531100 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.531071 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.531309 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.531293 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.531444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.531425 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.584662 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.584606 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.584891 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.584873 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4nk\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.585168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.585139 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hc9h\" (UniqueName: \"kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h\") pod \"maas-api-revoke-sub-f3ad57bc-42pdd\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.605921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.605901 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:03.615394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.615369 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:03.625336 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:03.625294 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" podStartSLOduration=1.625281626 podStartE2EDuration="1.625281626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:03.62227951 +0000 UTC m=+1184.224420654" watchObservedRunningTime="2026-10-31 18:41:03.625281626 +0000 UTC m=+1184.227422793" Oct 31 18:41:04.036789 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.036733 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd"] Oct 31 18:41:04.039008 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:04.038981 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb956115_e3d0_4b10_b0d9_b116b12f809e.slice/crio-33ddff4669bd8efb0e5ddf20a8e0fea167f176bbaf6502e59af045f3af98d025 WatchSource:0}: Error finding container 33ddff4669bd8efb0e5ddf20a8e0fea167f176bbaf6502e59af045f3af98d025: Status 404 returned error can't find the container with id 33ddff4669bd8efb0e5ddf20a8e0fea167f176bbaf6502e59af045f3af98d025 Oct 31 18:41:04.040969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.040946 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8"] Oct 31 18:41:04.041545 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:04.041523 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98a2456_b714_49ad_9d8c_acffc22ab681.slice/crio-760d4142fc136b80103eaae5d47704180a16a8663b9c00cbccb55e33bf2e26bb WatchSource:0}: Error finding container 760d4142fc136b80103eaae5d47704180a16a8663b9c00cbccb55e33bf2e26bb: Status 404 returned error can't find the container with id 760d4142fc136b80103eaae5d47704180a16a8663b9c00cbccb55e33bf2e26bb Oct 31 18:41:04.043772 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.043742 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:04.043833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.043813 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:04.043880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.043852 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:04.467328 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.467295 5688 generic.go:356] "Generic (PLEG): container finished" podID="294ca253-4a8a-4f07-8081-5dc30166e590" containerID="40c1a43a4f8cfbd93c61aea93c8c522151f84058dd81cc5d6f5972d86c4bb6eb" exitCode=0 Oct 31 18:41:04.467802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.467398 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" event={"ID":"294ca253-4a8a-4f07-8081-5dc30166e590","Type":"ContainerDied","Data":"40c1a43a4f8cfbd93c61aea93c8c522151f84058dd81cc5d6f5972d86c4bb6eb"} Oct 31 18:41:04.469632 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.469601 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" event={"ID":"c98a2456-b714-49ad-9d8c-acffc22ab681","Type":"ContainerStarted","Data":"245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0"} Oct 31 18:41:04.469754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.469651 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" event={"ID":"c98a2456-b714-49ad-9d8c-acffc22ab681","Type":"ContainerStarted","Data":"760d4142fc136b80103eaae5d47704180a16a8663b9c00cbccb55e33bf2e26bb"} Oct 31 18:41:04.472949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.472907 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" event={"ID":"bb956115-e3d0-4b10-b0d9-b116b12f809e","Type":"ContainerStarted","Data":"57c2a86bbb063adcf08e22edce800c171015f61cbf30212140a75fc7812f5d58"} Oct 31 18:41:04.473046 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.472963 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" event={"ID":"bb956115-e3d0-4b10-b0d9-b116b12f809e","Type":"ContainerStarted","Data":"33ddff4669bd8efb0e5ddf20a8e0fea167f176bbaf6502e59af045f3af98d025"} Oct 31 18:41:04.606337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.606253 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:04.702294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.702241 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" podStartSLOduration=1.7022273700000001 podStartE2EDuration="1.70222737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:04.700687692 +0000 UTC m=+1185.302828836" watchObservedRunningTime="2026-10-31 18:41:04.70222737 +0000 UTC m=+1185.304368513" Oct 31 18:41:04.820378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:04.820328 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podStartSLOduration=1.820315082 podStartE2EDuration="1.820315082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:04.819614169 +0000 UTC m=+1185.421755417" watchObservedRunningTime="2026-10-31 18:41:04.820315082 +0000 UTC m=+1185.422456225" Oct 31 18:41:05.480169 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.480136 5688 generic.go:356] "Generic (PLEG): container finished" podID="bb956115-e3d0-4b10-b0d9-b116b12f809e" containerID="57c2a86bbb063adcf08e22edce800c171015f61cbf30212140a75fc7812f5d58" exitCode=0 Oct 31 18:41:05.480617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.480244 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" event={"ID":"bb956115-e3d0-4b10-b0d9-b116b12f809e","Type":"ContainerDied","Data":"57c2a86bbb063adcf08e22edce800c171015f61cbf30212140a75fc7812f5d58"} Oct 31 18:41:05.607711 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.607682 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 18:41:05.607870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.607736 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 18:41:05.676550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.676523 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:05.872903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.872873 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca\") pod \"294ca253-4a8a-4f07-8081-5dc30166e590\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " Oct 31 18:41:05.873122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.872923 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj\" (UniqueName: \"kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj\") pod \"294ca253-4a8a-4f07-8081-5dc30166e590\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " Oct 31 18:41:05.873122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.873099 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca\") pod \"294ca253-4a8a-4f07-8081-5dc30166e590\" (UID: \"294ca253-4a8a-4f07-8081-5dc30166e590\") " Oct 31 18:41:05.873406 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.873380 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca" pod "294ca253-4a8a-4f07-8081-5dc30166e590" (UID: "294ca253-4a8a-4f07-8081-5dc30166e590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:41:05.873633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.873583 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:05.875145 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.875126 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca" pod "294ca253-4a8a-4f07-8081-5dc30166e590" (UID: "294ca253-4a8a-4f07-8081-5dc30166e590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:41:05.875355 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.875340 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj" pod "294ca253-4a8a-4f07-8081-5dc30166e590" (UID: "294ca253-4a8a-4f07-8081-5dc30166e590"). InnerVolumeSpecName "kube-api-access-bk7dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:41:05.974765 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.974737 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ca253-4a8a-4f07-8081-5dc30166e590-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:05.974765 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:05.974764 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7dj\" (UniqueName: \"kubernetes.io/projected/294ca253-4a8a-4f07-8081-5dc30166e590-kube-api-access-bk7dj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:06.487551 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.487528 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" Oct 31 18:41:06.488027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.487567 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72" event={"ID":"294ca253-4a8a-4f07-8081-5dc30166e590","Type":"ContainerDied","Data":"2215daf562c0c69c814d39c00ed36221e5b32e7c7091d5a3544db27d9d72a71b"} Oct 31 18:41:06.488027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.487615 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2215daf562c0c69c814d39c00ed36221e5b32e7c7091d5a3544db27d9d72a71b" Oct 31 18:41:06.606960 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.606928 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 18:41:06.607115 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.606986 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 18:41:06.692119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.692095 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:06.781934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.781840 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h\" (UniqueName: \"kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h\") pod \"bb956115-e3d0-4b10-b0d9-b116b12f809e\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " Oct 31 18:41:06.781934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.781888 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca\") pod \"bb956115-e3d0-4b10-b0d9-b116b12f809e\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " Oct 31 18:41:06.782522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.782488 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca" pod "bb956115-e3d0-4b10-b0d9-b116b12f809e" (UID: "bb956115-e3d0-4b10-b0d9-b116b12f809e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:41:06.784630 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.784605 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h" pod "bb956115-e3d0-4b10-b0d9-b116b12f809e" (UID: "bb956115-e3d0-4b10-b0d9-b116b12f809e"). InnerVolumeSpecName "kube-api-access-2hc9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:41:06.882982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.882952 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca\") pod \"bb956115-e3d0-4b10-b0d9-b116b12f809e\" (UID: \"bb956115-e3d0-4b10-b0d9-b116b12f809e\") " Oct 31 18:41:06.883263 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.883247 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hc9h\" (UniqueName: \"kubernetes.io/projected/bb956115-e3d0-4b10-b0d9-b116b12f809e-kube-api-access-2hc9h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:06.883366 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.883264 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:06.885112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.885089 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca" pod "bb956115-e3d0-4b10-b0d9-b116b12f809e" (UID: "bb956115-e3d0-4b10-b0d9-b116b12f809e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:41:06.984046 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:06.984014 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb956115-e3d0-4b10-b0d9-b116b12f809e-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:07.058256 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.057359 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h"] Oct 31 18:41:07.063232 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.061405 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb83ce8-wzd5h"] Oct 31 18:41:07.513210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.512959 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" event={"ID":"bb956115-e3d0-4b10-b0d9-b116b12f809e","Type":"ContainerDied","Data":"33ddff4669bd8efb0e5ddf20a8e0fea167f176bbaf6502e59af045f3af98d025"} Oct 31 18:41:07.513210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.513020 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ddff4669bd8efb0e5ddf20a8e0fea167f176bbaf6502e59af045f3af98d025" Oct 31 18:41:07.513210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.513086 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd" Oct 31 18:41:07.567379 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.567348 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc" path="/var/lib/kubelet/pods/5c5d8c02-c661-41e5-bdd4-8bfd52b1d0cc/volumes" Oct 31 18:41:07.606769 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.606740 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 18:41:07.606925 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:07.606788 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 18:41:08.606612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:08.606578 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 18:41:08.607070 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:08.606636 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 18:41:09.607221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:09.607173 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 18:41:09.607647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:09.607242 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 18:41:11.607243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:11.607197 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:41:11.607760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:11.607270 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:12.650486 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:12.650451 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:41:12.651076 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:12.651045 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:13.221550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:13.221520 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:13.222029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:13.222002 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:13.222978 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:13.222953 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:41:15.689092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.689061 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl"] Oct 31 18:41:15.690274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.690246 5688 cpu_manager.go:414] "containerMap: removing container" podUID="294ca253-4a8a-4f07-8081-5dc30166e590" containerName="revoke-keys" Oct 31 18:41:15.690274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.690269 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294ca253-4a8a-4f07-8081-5dc30166e590" containerName="revoke-keys" Oct 31 18:41:15.690427 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.690287 5688 cpu_manager.go:414] "containerMap: removing container" podUID="bb956115-e3d0-4b10-b0d9-b116b12f809e" containerName="revoke-keys" Oct 31 18:41:15.690427 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.690294 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb956115-e3d0-4b10-b0d9-b116b12f809e" containerName="revoke-keys" Oct 31 18:41:15.690527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.690447 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="294ca253-4a8a-4f07-8081-5dc30166e590" containerName="revoke-keys" Oct 31 18:41:15.690527 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.690461 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb956115-e3d0-4b10-b0d9-b116b12f809e" containerName="revoke-keys" Oct 31 18:41:15.694570 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.694552 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.796524 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.796494 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"distinct-2-w2-88085e-kserve-self-signed-certs\"" Oct 31 18:41:15.803217 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.800616 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl"] Oct 31 18:41:15.846961 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.846918 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.847329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.847297 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.857206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.847626 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.858253 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.858228 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5t8x\" (UniqueName: \"kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.858465 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.858443 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.892205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.858690 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988085 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988325 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988132 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988325 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988153 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988325 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988263 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988325 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988289 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5t8x\" (UniqueName: \"kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988332 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988599 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988649 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.988854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.988835 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.991470 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.991444 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:15.991683 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:15.991666 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:16.045014 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:16.044985 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5t8x\" (UniqueName: \"kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x\") pod \"distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:16.306905 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:16.306823 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:16.681215 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:16.681171 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl"] Oct 31 18:41:16.682925 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:16.682899 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03854b8e_b51c_4f0a_9126_23b1bf099a15.slice/crio-384912f63a30a4dd41af98f1f1c9b8e7619c3f5721699222e5bfbf66d57ecead WatchSource:0}: Error finding container 384912f63a30a4dd41af98f1f1c9b8e7619c3f5721699222e5bfbf66d57ecead: Status 404 returned error can't find the container with id 384912f63a30a4dd41af98f1f1c9b8e7619c3f5721699222e5bfbf66d57ecead Oct 31 18:41:17.585148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:17.585115 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" event={"ID":"03854b8e-b51c-4f0a-9126-23b1bf099a15","Type":"ContainerStarted","Data":"fd83f9137e779293350321abac7bdb2218691c642a228b76780be21082d00bbe"} Oct 31 18:41:17.585148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:17.585159 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" event={"ID":"03854b8e-b51c-4f0a-9126-23b1bf099a15","Type":"ContainerStarted","Data":"384912f63a30a4dd41af98f1f1c9b8e7619c3f5721699222e5bfbf66d57ecead"} Oct 31 18:41:17.699469 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:17.699422 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podStartSLOduration=2.699409005 podStartE2EDuration="2.699409005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:17.697413446 +0000 UTC m=+1198.299554589" watchObservedRunningTime="2026-10-31 18:41:17.699409005 +0000 UTC m=+1198.301550149" Oct 31 18:41:18.307561 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:18.307535 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:19.689726 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:19.689700 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:41:19.693954 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:19.693932 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:41:20.924080 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:20.924049 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7"] Oct 31 18:41:20.928819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:20.928800 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:20.947563 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:20.947538 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:20.947701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:20.947590 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:20.947798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:20.947777 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7xl\" (UniqueName: \"kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:20.981249 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:20.981217 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:41:21.031097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.031066 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:41:21.033674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.033653 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7"] Oct 31 18:41:21.049009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.048974 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7xl\" (UniqueName: \"kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.049105 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.049069 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.049105 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.049094 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.049804 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.049786 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.051857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.051839 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.105895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.105868 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr"] Oct 31 18:41:21.107969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.107945 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7xl\" (UniqueName: \"kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl\") pod \"maas-api-revoke-sub-0ceadf7c-vlvn7\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.110082 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.110063 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.145228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.145202 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx"] Oct 31 18:41:21.149087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.149065 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.150152 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150119 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150279 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150253 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkmr\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150367 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150349 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150415 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150401 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150467 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150430 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150610 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150574 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150672 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150746 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150730 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.150856 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.150833 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.171165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.171141 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr"] Oct 31 18:41:21.234738 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.234257 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx"] Oct 31 18:41:21.241995 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.241969 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.252717 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.252843 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.253387 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.253449 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.253826 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qr9c\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254135 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254199 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254251 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.254354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254300 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254811 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.254928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254835 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkmr\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254628 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254766 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254767 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.254928 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254864 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.255254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.254994 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.255254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.255038 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.255254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.255103 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.255457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.255434 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.255523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.255451 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.255523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.255481 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.255523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.255512 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.256135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.256111 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.256990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.256967 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.258318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.258218 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.317328 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.317291 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkmr\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.322476 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.321234 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.356877 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.356841 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357016 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.356916 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357016 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.356977 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357016 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357003 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qr9c\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357130 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357062 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357130 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357119 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357200 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357233 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357319 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357589 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357671 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.357857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.357826 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.358262 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.358240 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.358358 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.358332 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.359607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.359589 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.360118 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.360101 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.413671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.413645 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.413795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.413774 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qr9c\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.433975 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.433950 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:21.461833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.461807 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:21.628871 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.628847 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7"] Oct 31 18:41:21.630602 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:21.630573 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc173ab50_b591_422f_9025_bd1a76a90af5.slice/crio-88682b89abb8657282ff01350266e2fd1e5191f2737f0b6197f53c596627157d WatchSource:0}: Error finding container 88682b89abb8657282ff01350266e2fd1e5191f2737f0b6197f53c596627157d: Status 404 returned error can't find the container with id 88682b89abb8657282ff01350266e2fd1e5191f2737f0b6197f53c596627157d Oct 31 18:41:21.824290 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.824266 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr"] Oct 31 18:41:21.825120 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:21.825088 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8568b17_5b99_416e_81f7_ee05f6a471ef.slice/crio-cfea9614d6a4b60b7ebe11193c32067fb26ea8fd47b036afe6c9a6abdc44369e WatchSource:0}: Error finding container cfea9614d6a4b60b7ebe11193c32067fb26ea8fd47b036afe6c9a6abdc44369e: Status 404 returned error can't find the container with id cfea9614d6a4b60b7ebe11193c32067fb26ea8fd47b036afe6c9a6abdc44369e Oct 31 18:41:21.827521 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.827491 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:21.827596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.827556 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:21.827596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.827588 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:21.863554 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.863528 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx"] Oct 31 18:41:21.864455 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:21.864433 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b235b24_51ac_406e_9f42_6d3196faf32a.slice/crio-5660cdcc328c3a4c5c54d9ea4a3b8b52a034aa660ea355fe50f58e30d946cd85 WatchSource:0}: Error finding container 5660cdcc328c3a4c5c54d9ea4a3b8b52a034aa660ea355fe50f58e30d946cd85: Status 404 returned error can't find the container with id 5660cdcc328c3a4c5c54d9ea4a3b8b52a034aa660ea355fe50f58e30d946cd85 Oct 31 18:41:21.867044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.867016 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:21.867108 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.867085 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:21.867151 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:21.867115 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:41:22.616612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.616468 5688 generic.go:356] "Generic (PLEG): container finished" podID="c173ab50-b591-422f-9025-bd1a76a90af5" containerID="da4cb3798e33e7ccb6a598c795920846d8423caf31c76a4349813aa4d4d777c5" exitCode=0 Oct 31 18:41:22.616612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.616539 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" event={"ID":"c173ab50-b591-422f-9025-bd1a76a90af5","Type":"ContainerDied","Data":"da4cb3798e33e7ccb6a598c795920846d8423caf31c76a4349813aa4d4d777c5"} Oct 31 18:41:22.616612 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.616588 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" event={"ID":"c173ab50-b591-422f-9025-bd1a76a90af5","Type":"ContainerStarted","Data":"88682b89abb8657282ff01350266e2fd1e5191f2737f0b6197f53c596627157d"} Oct 31 18:41:22.618285 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.618266 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" event={"ID":"5b235b24-51ac-406e-9f42-6d3196faf32a","Type":"ContainerStarted","Data":"4414ed8c2098c939df182a13a93ffedba06e2279a07f8e7951b77c6c0b9ad88a"} Oct 31 18:41:22.618364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.618305 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" event={"ID":"5b235b24-51ac-406e-9f42-6d3196faf32a","Type":"ContainerStarted","Data":"5660cdcc328c3a4c5c54d9ea4a3b8b52a034aa660ea355fe50f58e30d946cd85"} Oct 31 18:41:22.619920 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.619900 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" event={"ID":"b8568b17-5b99-416e-81f7-ee05f6a471ef","Type":"ContainerStarted","Data":"4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54"} Oct 31 18:41:22.619986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:22.619938 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" event={"ID":"b8568b17-5b99-416e-81f7-ee05f6a471ef","Type":"ContainerStarted","Data":"cfea9614d6a4b60b7ebe11193c32067fb26ea8fd47b036afe6c9a6abdc44369e"} Oct 31 18:41:23.016255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.016199 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podStartSLOduration=2.016167475 podStartE2EDuration="2.016167475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:23.014356466 +0000 UTC m=+1203.616497610" watchObservedRunningTime="2026-10-31 18:41:23.016167475 +0000 UTC m=+1203.618308618" Oct 31 18:41:23.016755 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.016726 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podStartSLOduration=2.016716418 podStartE2EDuration="2.016716418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:22.881683423 +0000 UTC m=+1203.483824568" watchObservedRunningTime="2026-10-31 18:41:23.016716418 +0000 UTC m=+1203.618857568" Oct 31 18:41:23.070209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.069236 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc"] Oct 31 18:41:23.086208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.086149 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103a940f-xmlgc"] Oct 31 18:41:23.443373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.436454 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:23.447208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.446294 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 18:41:23.447208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.446353 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 18:41:23.470909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.470878 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 18:41:23.471166 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.471134 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 18:41:23.490352 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.490270 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:23.587205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.586905 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb" path="/var/lib/kubelet/pods/1cf240f5-a75e-47a9-8a8d-e60e0b6e7acb/volumes" Oct 31 18:41:23.918504 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:23.918483 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:24.011049 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.011015 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl\" (UniqueName: \"kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl\") pod \"c173ab50-b591-422f-9025-bd1a76a90af5\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " Oct 31 18:41:24.011222 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.011061 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca\") pod \"c173ab50-b591-422f-9025-bd1a76a90af5\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " Oct 31 18:41:24.011222 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.011141 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca\") pod \"c173ab50-b591-422f-9025-bd1a76a90af5\" (UID: \"c173ab50-b591-422f-9025-bd1a76a90af5\") " Oct 31 18:41:24.011480 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.011431 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca" pod "c173ab50-b591-422f-9025-bd1a76a90af5" (UID: "c173ab50-b591-422f-9025-bd1a76a90af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:41:24.011613 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.011532 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:24.013591 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.013563 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl" pod "c173ab50-b591-422f-9025-bd1a76a90af5" (UID: "c173ab50-b591-422f-9025-bd1a76a90af5"). InnerVolumeSpecName "kube-api-access-nt7xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:41:24.014467 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.014447 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca" pod "c173ab50-b591-422f-9025-bd1a76a90af5" (UID: "c173ab50-b591-422f-9025-bd1a76a90af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:41:24.112395 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.112366 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173ab50-b591-422f-9025-bd1a76a90af5-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:24.112395 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.112394 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt7xl\" (UniqueName: \"kubernetes.io/projected/c173ab50-b591-422f-9025-bd1a76a90af5-kube-api-access-nt7xl\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:24.437198 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.436501 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 18:41:24.437198 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.436556 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 18:41:24.464805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.464533 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 18:41:24.464805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.464585 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 18:41:24.688274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.684335 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" Oct 31 18:41:24.688274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.685880 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7" event={"ID":"c173ab50-b591-422f-9025-bd1a76a90af5","Type":"ContainerDied","Data":"88682b89abb8657282ff01350266e2fd1e5191f2737f0b6197f53c596627157d"} Oct 31 18:41:24.688274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.685930 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88682b89abb8657282ff01350266e2fd1e5191f2737f0b6197f53c596627157d" Oct 31 18:41:24.690918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.690894 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp"] Oct 31 18:41:24.691613 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.691596 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c173ab50-b591-422f-9025-bd1a76a90af5" containerName="revoke-keys" Oct 31 18:41:24.691664 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.691617 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c173ab50-b591-422f-9025-bd1a76a90af5" containerName="revoke-keys" Oct 31 18:41:24.691786 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.691774 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c173ab50-b591-422f-9025-bd1a76a90af5" containerName="revoke-keys" Oct 31 18:41:24.698488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.698470 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.720367 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.720339 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42hj\" (UniqueName: \"kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.720499 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.720445 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.720499 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.720475 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.723059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.723035 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp"] Oct 31 18:41:24.756500 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.756471 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:41:24.780208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.757640 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:41:24.821392 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.821359 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42hj\" (UniqueName: \"kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.821540 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.821421 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.821540 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.821447 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.822043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.822021 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.824518 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.824501 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.888967 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:24.888915 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42hj\" (UniqueName: \"kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj\") pod \"maas-api-revoke-sub-9a632e73-rp9zp\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:24.912726 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:41:24.912696 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:41:25.014645 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.014617 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:25.456268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.455282 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 18:41:25.456268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.455359 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 18:41:25.466582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.466552 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 18:41:25.466836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.466803 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 18:41:25.486947 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.486916 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp"] Oct 31 18:41:25.492219 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:25.492174 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f97e7ab_dbac_417b_89f3_5b5f598286cd.slice/crio-bf41fcf24e578100f2e6f524fda408b4ce70d6342c0644b9e611043f97aee0d0 WatchSource:0}: Error finding container bf41fcf24e578100f2e6f524fda408b4ce70d6342c0644b9e611043f97aee0d0: Status 404 returned error can't find the container with id bf41fcf24e578100f2e6f524fda408b4ce70d6342c0644b9e611043f97aee0d0 Oct 31 18:41:25.701153 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:25.701107 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" event={"ID":"1f97e7ab-dbac-417b-89f3-5b5f598286cd","Type":"ContainerStarted","Data":"bf41fcf24e578100f2e6f524fda408b4ce70d6342c0644b9e611043f97aee0d0"} Oct 31 18:41:26.307261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.307218 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:26.319914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.319892 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:26.468208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.467320 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 18:41:26.468208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.467380 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 18:41:26.468208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.468056 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 18:41:26.468208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.468092 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 18:41:26.709962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.709933 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f97e7ab-dbac-417b-89f3-5b5f598286cd" containerID="35af03604280eab54b17ca0da81967e903ed6c0003819a37e92ce3cb8ad8d251" exitCode=0 Oct 31 18:41:26.710174 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.710145 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" event={"ID":"1f97e7ab-dbac-417b-89f3-5b5f598286cd","Type":"ContainerDied","Data":"35af03604280eab54b17ca0da81967e903ed6c0003819a37e92ce3cb8ad8d251"} Oct 31 18:41:26.721423 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:26.721391 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:27.438490 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:27.438455 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 18:41:27.439074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:27.439046 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 18:41:27.467130 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:27.467098 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 18:41:27.475049 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:27.475018 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 18:41:27.901996 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:27.901971 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:28.020299 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.020264 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca\") pod \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " Oct 31 18:41:28.020535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.020513 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj\" (UniqueName: \"kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj\") pod \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " Oct 31 18:41:28.020622 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.020608 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca\") pod \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\" (UID: \"1f97e7ab-dbac-417b-89f3-5b5f598286cd\") " Oct 31 18:41:28.023992 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.023955 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca" pod "1f97e7ab-dbac-417b-89f3-5b5f598286cd" (UID: "1f97e7ab-dbac-417b-89f3-5b5f598286cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:41:28.024360 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.024336 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca" pod "1f97e7ab-dbac-417b-89f3-5b5f598286cd" (UID: "1f97e7ab-dbac-417b-89f3-5b5f598286cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:41:28.027012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.026987 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj" pod "1f97e7ab-dbac-417b-89f3-5b5f598286cd" (UID: "1f97e7ab-dbac-417b-89f3-5b5f598286cd"). InnerVolumeSpecName "kube-api-access-x42hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:41:28.121986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.121953 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:28.121986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.121980 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42hj\" (UniqueName: \"kubernetes.io/projected/1f97e7ab-dbac-417b-89f3-5b5f598286cd-kube-api-access-x42hj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:28.121986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.121992 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f97e7ab-dbac-417b-89f3-5b5f598286cd-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:28.434750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.434662 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 18:41:28.434750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.434714 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 18:41:28.463073 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.463038 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 18:41:28.463551 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.463080 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 18:41:28.728253 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.728115 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" event={"ID":"1f97e7ab-dbac-417b-89f3-5b5f598286cd","Type":"ContainerDied","Data":"bf41fcf24e578100f2e6f524fda408b4ce70d6342c0644b9e611043f97aee0d0"} Oct 31 18:41:28.728253 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.728146 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp" Oct 31 18:41:28.728253 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:28.728195 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf41fcf24e578100f2e6f524fda408b4ce70d6342c0644b9e611043f97aee0d0" Oct 31 18:41:29.048093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.048061 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm"] Oct 31 18:41:29.052463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.052434 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08b7312-gxxcm"] Oct 31 18:41:29.438586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.438522 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:29.438843 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.438828 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:29.439464 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.439446 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:41:29.466456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.466436 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:29.466785 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.466695 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:29.467357 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.467342 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:41:29.566350 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:29.566316 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4708701b-35ae-469f-bad0-63cbf0417e14" path="/var/lib/kubelet/pods/4708701b-35ae-469f-bad0-63cbf0417e14/volumes" Oct 31 18:41:31.660890 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.660860 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl"] Oct 31 18:41:31.661365 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.661340 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f97e7ab-dbac-417b-89f3-5b5f598286cd" containerName="revoke-keys" Oct 31 18:41:31.661365 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.661366 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f97e7ab-dbac-417b-89f3-5b5f598286cd" containerName="revoke-keys" Oct 31 18:41:31.661465 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.661454 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f97e7ab-dbac-417b-89f3-5b5f598286cd" containerName="revoke-keys" Oct 31 18:41:31.665922 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.665906 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.724031 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.723548 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:41:31.749822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.746259 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl"] Oct 31 18:41:31.764998 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.764965 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.765579 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.765548 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.768315 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.768289 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwsg\" (UniqueName: \"kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.805438 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.805369 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:41:31.869155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.869126 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.869155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.869159 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwsg\" (UniqueName: \"kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.869400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.869269 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.870344 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.870290 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.879710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.879681 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.926579 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.926520 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwsg\" (UniqueName: \"kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg\") pod \"maas-api-revoke-sub-00598967-rxfwl\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:31.979266 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:31.979241 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:32.350613 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:32.350589 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl"] Oct 31 18:41:32.352366 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:41:32.352331 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457f98ce_f1dd_4e01_a7a8_1148df58d77a.slice/crio-6e94022fad7c3c1885facdae54ea2fa6f5b02e23158af5f69e16f67699660107 WatchSource:0}: Error finding container 6e94022fad7c3c1885facdae54ea2fa6f5b02e23158af5f69e16f67699660107: Status 404 returned error can't find the container with id 6e94022fad7c3c1885facdae54ea2fa6f5b02e23158af5f69e16f67699660107 Oct 31 18:41:32.853206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:32.848866 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" event={"ID":"457f98ce-f1dd-4e01-a7a8-1148df58d77a","Type":"ContainerStarted","Data":"2652e7260ab428bbc9b6673e0953f70b627060954dbf86a58c3f7b0972348436"} Oct 31 18:41:32.853206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:32.848933 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" event={"ID":"457f98ce-f1dd-4e01-a7a8-1148df58d77a","Type":"ContainerStarted","Data":"6e94022fad7c3c1885facdae54ea2fa6f5b02e23158af5f69e16f67699660107"} Oct 31 18:41:33.878087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:33.878051 5688 generic.go:356] "Generic (PLEG): container finished" podID="457f98ce-f1dd-4e01-a7a8-1148df58d77a" containerID="2652e7260ab428bbc9b6673e0953f70b627060954dbf86a58c3f7b0972348436" exitCode=0 Oct 31 18:41:33.889265 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:33.889230 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" event={"ID":"457f98ce-f1dd-4e01-a7a8-1148df58d77a","Type":"ContainerDied","Data":"2652e7260ab428bbc9b6673e0953f70b627060954dbf86a58c3f7b0972348436"} Oct 31 18:41:34.286209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:34.285358 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:34.660272 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:34.659239 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 18:41:34.717207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:34.716449 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 18:41:34.801208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:34.800276 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:34.944209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:34.943288 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.060207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.059282 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 31 18:41:35.178264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.162832 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" podUID="3c10670a-933d-4f20-a46d-cc4cb63e7091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.296306 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.294124 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.296306 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.294788 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:41:35.455567 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.454547 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.501448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.501292 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:41:35.501448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.501356 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.779219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.773440 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 18:41:35.779219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.773896 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.779219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.775046 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:35.848472 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.848336 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Oct 31 18:41:35.926268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.926032 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 18:41:35.969971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:35.947551 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:36.069365 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.068822 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260726 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260793 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260829 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260863 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260897 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260929 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 18:41:36.303410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.260961 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 18:41:36.309209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.305824 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:41:36.309463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.309434 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:36.397084 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.396302 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 18:41:36.407495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.407471 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:36.461403 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.461336 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 18:41:36.466056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.461718 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:41:36.714259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.598619 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg\" (UniqueName: \"kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg\") pod \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " Oct 31 18:41:36.714259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.621372 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca\") pod \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " Oct 31 18:41:36.714259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.621519 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca\") pod \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\" (UID: \"457f98ce-f1dd-4e01-a7a8-1148df58d77a\") " Oct 31 18:41:36.714259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.696105 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca" pod "457f98ce-f1dd-4e01-a7a8-1148df58d77a" (UID: "457f98ce-f1dd-4e01-a7a8-1148df58d77a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:41:36.751211 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.740110 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:36.751927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.751892 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca" pod "457f98ce-f1dd-4e01-a7a8-1148df58d77a" (UID: "457f98ce-f1dd-4e01-a7a8-1148df58d77a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:41:36.776498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.776454 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:36.832721 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.832677 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:36.833462 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.833431 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg" pod "457f98ce-f1dd-4e01-a7a8-1148df58d77a" (UID: "457f98ce-f1dd-4e01-a7a8-1148df58d77a"). InnerVolumeSpecName "kube-api-access-xjwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:41:36.866472 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.866433 5688 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:36.877688 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.877634 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457f98ce-f1dd-4e01-a7a8-1148df58d77a-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:36.877966 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.877948 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjwsg\" (UniqueName: \"kubernetes.io/projected/457f98ce-f1dd-4e01-a7a8-1148df58d77a-kube-api-access-xjwsg\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:41:36.942732 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.942683 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:36.947347 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.947315 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:41:36.965808 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:36.965719 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:37.005610 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.005570 5688 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:37.077545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.077517 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 18:41:37.342796 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.342757 5688 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 31 18:41:37.462210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.461696 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:37.515558 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.515525 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" Oct 31 18:41:37.586833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.554377 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl" event={"ID":"457f98ce-f1dd-4e01-a7a8-1148df58d77a","Type":"ContainerDied","Data":"6e94022fad7c3c1885facdae54ea2fa6f5b02e23158af5f69e16f67699660107"} Oct 31 18:41:37.586833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.554465 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e94022fad7c3c1885facdae54ea2fa6f5b02e23158af5f69e16f67699660107" Oct 31 18:41:37.773341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.773264 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:37.792295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.792255 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:37.806876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.806831 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Oct 31 18:41:37.833218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.833154 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:37.965803 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.965777 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 18:41:37.967217 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:37.967168 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:38.333237 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:38.332150 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 18:41:38.384243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:38.383139 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 18:41:38.461862 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:38.460579 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 18:41:38.642281 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:38.642174 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 18:41:40.055209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:40.055157 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59"] Oct 31 18:41:40.097165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:40.097137 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376095ff-flt59"] Oct 31 18:41:41.565636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:41.565604 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dc18d1-6681-4a97-bcb5-d62469ce935c" path="/var/lib/kubelet/pods/e7dc18d1-6681-4a97-bcb5-d62469ce935c/volumes" Oct 31 18:41:48.045586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:48.045553 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw"] Oct 31 18:41:48.094974 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:48.094946 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b4b4b6-qwhlw"] Oct 31 18:41:49.566535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:49.566505 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe16175-8a01-4e28-b92c-3f96ae311773" path="/var/lib/kubelet/pods/fbe16175-8a01-4e28-b92c-3f96ae311773/volumes" Oct 31 18:41:54.903556 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:41:54.903515 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:41:58.397071 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.397031 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7"] Oct 31 18:41:58.398944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.398914 5688 cpu_manager.go:414] "containerMap: removing container" podUID="457f98ce-f1dd-4e01-a7a8-1148df58d77a" containerName="revoke-keys" Oct 31 18:41:58.399111 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.399096 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457f98ce-f1dd-4e01-a7a8-1148df58d77a" containerName="revoke-keys" Oct 31 18:41:58.399565 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.399546 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="457f98ce-f1dd-4e01-a7a8-1148df58d77a" containerName="revoke-keys" Oct 31 18:41:58.422415 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.422389 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537252 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537331 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537359 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537466 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537545 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6fs\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537715 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537746 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537824 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.546209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.537895 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.607237 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.599872 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7"] Oct 31 18:41:58.689255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.665679 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.711440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.711414 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.711659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.711639 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.711795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.711778 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.712017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.711998 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.712158 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.712141 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.712313 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.712296 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.712455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.712440 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.712586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.712568 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6fs\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.741388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.714837 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.741388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.715090 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.741388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.738455 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.741388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.741087 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.741388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.741344 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.829318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.829245 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.893981 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.892879 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6fs\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.958070 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.957991 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:58.985737 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:58.985671 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:59.058593 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:59.058345 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:41:59.784226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:41:59.784143 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 18:42:00.037254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:00.034098 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7"] Oct 31 18:42:00.059258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:00.059209 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:42:00.059535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:00.059509 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:42:00.059752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:00.059723 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:42:00.993076 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:00.992046 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" event={"ID":"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c","Type":"ContainerStarted","Data":"3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3"} Oct 31 18:42:00.993076 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:00.992106 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" event={"ID":"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c","Type":"ContainerStarted","Data":"cad14aa0ae9f8087449d888d85c5efc99e2c431a9073607b6edf34fb1ab4c9c7"} Oct 31 18:42:01.060208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:01.059651 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:42:01.061467 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:01.061147 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:01.061467 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:01.061216 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:01.180209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:01.178210 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podStartSLOduration=3.178168952 podStartE2EDuration="3.178168952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:01.1606702 +0000 UTC m=+1241.762811345" watchObservedRunningTime="2026-10-31 18:42:01.178168952 +0000 UTC m=+1241.780310096" Oct 31 18:42:01.578851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:01.569146 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lqzg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:42:01.578851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:01.569239 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:42:02.059641 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:02.059605 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:02.060087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:02.059656 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:03.059671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:03.059612 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:03.060055 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:03.059678 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:04.059614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:04.059582 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:04.059788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:04.059634 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:05.059845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:05.059527 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:05.059845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:05.059579 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:06.060220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:06.059947 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:06.060220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:06.060008 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:07.060092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:07.060062 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 18:42:07.060278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:07.060107 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 18:42:08.107537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.107498 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:42:08.108719 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.108702 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:42:08.110154 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.110125 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:42:08.875450 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.875412 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr"] Oct 31 18:42:08.879448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.879432 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:08.933110 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.933051 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:42:08.933293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.933234 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:42:08.936961 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.936926 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr"] Oct 31 18:42:08.943902 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.943875 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:08.944117 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.944093 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8g9h\" (UniqueName: \"kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:08.944279 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:08.944148 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.045017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.044987 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.045164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.045057 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.045164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.045124 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8g9h\" (UniqueName: \"kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.045625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.045591 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.047457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.047438 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.102477 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.102451 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8g9h\" (UniqueName: \"kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h\") pod \"maas-api-revoke-sub-786aef1f-6cwtr\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.192535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.192468 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:09.568652 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:09.568622 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr"] Oct 31 18:42:09.570042 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:42:09.570020 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b9ea6f_c068_4fa5_86bc_165e15c85af7.slice/crio-6d85b786f179cf93773bf369e14704ccab5c66c96a3fcff80f446624f7529ed8 WatchSource:0}: Error finding container 6d85b786f179cf93773bf369e14704ccab5c66c96a3fcff80f446624f7529ed8: Status 404 returned error can't find the container with id 6d85b786f179cf93773bf369e14704ccab5c66c96a3fcff80f446624f7529ed8 Oct 31 18:42:10.052268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:10.052235 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" event={"ID":"26b9ea6f-c068-4fa5-86bc-165e15c85af7","Type":"ContainerStarted","Data":"b29b3e77225d76fcc8551f1420cb9f1947b9238271e2c751ae2d6cb9095209ee"} Oct 31 18:42:10.052416 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:10.052286 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" event={"ID":"26b9ea6f-c068-4fa5-86bc-165e15c85af7","Type":"ContainerStarted","Data":"6d85b786f179cf93773bf369e14704ccab5c66c96a3fcff80f446624f7529ed8"} Oct 31 18:42:10.165694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:10.165606 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" podStartSLOduration=2.165592193 podStartE2EDuration="2.165592193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:10.163789156 +0000 UTC m=+1250.765930301" watchObservedRunningTime="2026-10-31 18:42:10.165592193 +0000 UTC m=+1250.767733358" Oct 31 18:42:11.060309 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:11.060270 5688 generic.go:356] "Generic (PLEG): container finished" podID="26b9ea6f-c068-4fa5-86bc-165e15c85af7" containerID="b29b3e77225d76fcc8551f1420cb9f1947b9238271e2c751ae2d6cb9095209ee" exitCode=0 Oct 31 18:42:11.060679 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:11.060351 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" event={"ID":"26b9ea6f-c068-4fa5-86bc-165e15c85af7","Type":"ContainerDied","Data":"b29b3e77225d76fcc8551f1420cb9f1947b9238271e2c751ae2d6cb9095209ee"} Oct 31 18:42:12.208619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.208593 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:12.289724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.289697 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h\" (UniqueName: \"kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h\") pod \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " Oct 31 18:42:12.289858 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.289823 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca\") pod \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " Oct 31 18:42:12.289901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.289857 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca\") pod \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\" (UID: \"26b9ea6f-c068-4fa5-86bc-165e15c85af7\") " Oct 31 18:42:12.290254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.290231 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca" pod "26b9ea6f-c068-4fa5-86bc-165e15c85af7" (UID: "26b9ea6f-c068-4fa5-86bc-165e15c85af7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:42:12.291865 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.291838 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h" pod "26b9ea6f-c068-4fa5-86bc-165e15c85af7" (UID: "26b9ea6f-c068-4fa5-86bc-165e15c85af7"). InnerVolumeSpecName "kube-api-access-f8g9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:42:12.291950 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.291930 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca" pod "26b9ea6f-c068-4fa5-86bc-165e15c85af7" (UID: "26b9ea6f-c068-4fa5-86bc-165e15c85af7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:42:12.391143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.391076 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8g9h\" (UniqueName: \"kubernetes.io/projected/26b9ea6f-c068-4fa5-86bc-165e15c85af7-kube-api-access-f8g9h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:12.391143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.391102 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:12.391143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:12.391112 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b9ea6f-c068-4fa5-86bc-165e15c85af7-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:13.075590 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:13.075560 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" Oct 31 18:42:13.075760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:13.075581 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr" event={"ID":"26b9ea6f-c068-4fa5-86bc-165e15c85af7","Type":"ContainerDied","Data":"6d85b786f179cf93773bf369e14704ccab5c66c96a3fcff80f446624f7529ed8"} Oct 31 18:42:13.075760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:13.075634 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d85b786f179cf93773bf369e14704ccab5c66c96a3fcff80f446624f7529ed8" Oct 31 18:42:17.825694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.825663 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f"] Oct 31 18:42:17.826133 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.826121 5688 cpu_manager.go:414] "containerMap: removing container" podUID="26b9ea6f-c068-4fa5-86bc-165e15c85af7" containerName="revoke-keys" Oct 31 18:42:17.826208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.826134 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b9ea6f-c068-4fa5-86bc-165e15c85af7" containerName="revoke-keys" Oct 31 18:42:17.826253 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.826241 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b9ea6f-c068-4fa5-86bc-165e15c85af7" containerName="revoke-keys" Oct 31 18:42:17.833286 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.831172 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:17.885737 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.885712 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f"] Oct 31 18:42:17.931851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.931823 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:42:17.931851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.931820 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:42:17.961586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.961562 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plb22\" (UniqueName: \"kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:17.961704 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.961676 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:17.961767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:17.961752 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.062516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.062494 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.062641 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.062609 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.062715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.062701 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb22\" (UniqueName: \"kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.063236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.063144 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.064864 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.064844 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.118981 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.118920 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb22\" (UniqueName: \"kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22\") pod \"maas-api-revoke-sub-3ccf13ff-9wb7f\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.148296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.148272 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:18.521900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:18.521875 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f"] Oct 31 18:42:18.523974 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:42:18.523950 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d83f600_e654_4da1_bb65_c4ff81f0c38d.slice/crio-4fb732b3d093bb2422e2096d2ef78ddf29335846c3d034a97b85c80abbd888c0 WatchSource:0}: Error finding container 4fb732b3d093bb2422e2096d2ef78ddf29335846c3d034a97b85c80abbd888c0: Status 404 returned error can't find the container with id 4fb732b3d093bb2422e2096d2ef78ddf29335846c3d034a97b85c80abbd888c0 Oct 31 18:42:19.119903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:19.119815 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" event={"ID":"0d83f600-e654-4da1-bb65-c4ff81f0c38d","Type":"ContainerStarted","Data":"5609f4ba456c9bf06fa571e4586688f92ac421d32021fe73cd7a77e77d0466fe"} Oct 31 18:42:19.120473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:19.120439 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" event={"ID":"0d83f600-e654-4da1-bb65-c4ff81f0c38d","Type":"ContainerStarted","Data":"4fb732b3d093bb2422e2096d2ef78ddf29335846c3d034a97b85c80abbd888c0"} Oct 31 18:42:19.233088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:19.233038 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" podStartSLOduration=2.233022754 podStartE2EDuration="2.233022754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:19.231746219 +0000 UTC m=+1259.833887365" watchObservedRunningTime="2026-10-31 18:42:19.233022754 +0000 UTC m=+1259.835163897" Oct 31 18:42:19.706141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:19.706063 5688 scope.go:122] "RemoveContainer" containerID="9ec7c2dbc45dfc850c11b0115a1f8bb4c129555faa03e2e99a9e7a2f280430d6" Oct 31 18:42:19.715676 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:19.715648 5688 scope.go:122] "RemoveContainer" containerID="d473ce31682d2ca6c2df6a502593454176a1db155d49fbfa9203b38536a855e1" Oct 31 18:42:19.724621 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:19.724592 5688 scope.go:122] "RemoveContainer" containerID="7e0befc608172d4dfcd7490256b4e280e846f975a5592b84e018ad8d86ce6c18" Oct 31 18:42:20.127154 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:20.127122 5688 generic.go:356] "Generic (PLEG): container finished" podID="0d83f600-e654-4da1-bb65-c4ff81f0c38d" containerID="5609f4ba456c9bf06fa571e4586688f92ac421d32021fe73cd7a77e77d0466fe" exitCode=0 Oct 31 18:42:20.127578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:20.127215 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" event={"ID":"0d83f600-e654-4da1-bb65-c4ff81f0c38d","Type":"ContainerDied","Data":"5609f4ba456c9bf06fa571e4586688f92ac421d32021fe73cd7a77e77d0466fe"} Oct 31 18:42:21.418949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.418924 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:21.466167 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.466106 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22\" (UniqueName: \"kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22\") pod \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " Oct 31 18:42:21.466323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.466168 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca\") pod \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " Oct 31 18:42:21.466323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.466230 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca\") pod \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\" (UID: \"0d83f600-e654-4da1-bb65-c4ff81f0c38d\") " Oct 31 18:42:21.466530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.466497 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca" pod "0d83f600-e654-4da1-bb65-c4ff81f0c38d" (UID: "0d83f600-e654-4da1-bb65-c4ff81f0c38d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:42:21.466821 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.466803 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:21.468304 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.468282 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca" pod "0d83f600-e654-4da1-bb65-c4ff81f0c38d" (UID: "0d83f600-e654-4da1-bb65-c4ff81f0c38d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:42:21.468400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.468378 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22" pod "0d83f600-e654-4da1-bb65-c4ff81f0c38d" (UID: "0d83f600-e654-4da1-bb65-c4ff81f0c38d"). InnerVolumeSpecName "kube-api-access-plb22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:42:21.579176 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.579139 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d83f600-e654-4da1-bb65-c4ff81f0c38d-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:21.583267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:21.583242 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plb22\" (UniqueName: \"kubernetes.io/projected/0d83f600-e654-4da1-bb65-c4ff81f0c38d-kube-api-access-plb22\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:22.170893 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.170851 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" event={"ID":"0d83f600-e654-4da1-bb65-c4ff81f0c38d","Type":"ContainerDied","Data":"4fb732b3d093bb2422e2096d2ef78ddf29335846c3d034a97b85c80abbd888c0"} Oct 31 18:42:22.171044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.170915 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb732b3d093bb2422e2096d2ef78ddf29335846c3d034a97b85c80abbd888c0" Oct 31 18:42:22.171044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.170875 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f" Oct 31 18:42:22.584101 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.584068 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf"] Oct 31 18:42:22.584640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.584625 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0d83f600-e654-4da1-bb65-c4ff81f0c38d" containerName="revoke-keys" Oct 31 18:42:22.584686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.584643 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d83f600-e654-4da1-bb65-c4ff81f0c38d" containerName="revoke-keys" Oct 31 18:42:22.584742 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.584734 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d83f600-e654-4da1-bb65-c4ff81f0c38d" containerName="revoke-keys" Oct 31 18:42:22.588085 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.588067 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.608472 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.608450 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.608557 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.608513 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.608611 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.608595 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dp9\" (UniqueName: \"kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.642895 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.642870 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf"] Oct 31 18:42:22.688413 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.688380 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:42:22.688518 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.688416 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:42:22.710038 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.710014 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.710150 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.710077 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.710150 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.710097 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dp9\" (UniqueName: \"kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.710663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.710629 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.712566 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.712546 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.767073 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.767050 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dp9\" (UniqueName: \"kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9\") pod \"maas-api-revoke-sub-efb6bb94-kxnqf\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:22.902261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:22.901804 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:23.345037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:23.345008 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf"] Oct 31 18:42:23.346045 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:42:23.346020 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3c986b_530c_4205_8180_dd32b661530d.slice/crio-2bef53b570d12522df2bf0d2f08a147e84c1018bb9b854e8536c225efb8bbd4a WatchSource:0}: Error finding container 2bef53b570d12522df2bf0d2f08a147e84c1018bb9b854e8536c225efb8bbd4a: Status 404 returned error can't find the container with id 2bef53b570d12522df2bf0d2f08a147e84c1018bb9b854e8536c225efb8bbd4a Oct 31 18:42:24.184717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:24.184685 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" event={"ID":"cc3c986b-530c-4205-8180-dd32b661530d","Type":"ContainerStarted","Data":"74b8289b466ec024d45a10a20c2cda4ce972c704897576f51d66a8ebc86d6733"} Oct 31 18:42:24.185123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:24.184763 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" event={"ID":"cc3c986b-530c-4205-8180-dd32b661530d","Type":"ContainerStarted","Data":"2bef53b570d12522df2bf0d2f08a147e84c1018bb9b854e8536c225efb8bbd4a"} Oct 31 18:42:24.299195 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:24.299136 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" podStartSLOduration=2.29912401 podStartE2EDuration="2.29912401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:24.296487639 +0000 UTC m=+1264.898628784" watchObservedRunningTime="2026-10-31 18:42:24.29912401 +0000 UTC m=+1264.901265154" Oct 31 18:42:24.904101 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:42:24.904065 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:42:25.192602 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:25.192515 5688 generic.go:356] "Generic (PLEG): container finished" podID="cc3c986b-530c-4205-8180-dd32b661530d" containerID="74b8289b466ec024d45a10a20c2cda4ce972c704897576f51d66a8ebc86d6733" exitCode=0 Oct 31 18:42:25.192602 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:25.192569 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" event={"ID":"cc3c986b-530c-4205-8180-dd32b661530d","Type":"ContainerDied","Data":"74b8289b466ec024d45a10a20c2cda4ce972c704897576f51d66a8ebc86d6733"} Oct 31 18:42:26.359795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.359760 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:26.447768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.447739 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca\") pod \"cc3c986b-530c-4205-8180-dd32b661530d\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " Oct 31 18:42:26.447900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.447831 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9\" (UniqueName: \"kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9\") pod \"cc3c986b-530c-4205-8180-dd32b661530d\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " Oct 31 18:42:26.447900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.447891 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca\") pod \"cc3c986b-530c-4205-8180-dd32b661530d\" (UID: \"cc3c986b-530c-4205-8180-dd32b661530d\") " Oct 31 18:42:26.448108 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.448083 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca" pod "cc3c986b-530c-4205-8180-dd32b661530d" (UID: "cc3c986b-530c-4205-8180-dd32b661530d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:42:26.450039 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.449973 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca" pod "cc3c986b-530c-4205-8180-dd32b661530d" (UID: "cc3c986b-530c-4205-8180-dd32b661530d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:42:26.450039 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.449980 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9" pod "cc3c986b-530c-4205-8180-dd32b661530d" (UID: "cc3c986b-530c-4205-8180-dd32b661530d"). InnerVolumeSpecName "kube-api-access-v9dp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:42:26.549021 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.548989 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9dp9\" (UniqueName: \"kubernetes.io/projected/cc3c986b-530c-4205-8180-dd32b661530d-kube-api-access-v9dp9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:26.549021 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.549016 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:26.549021 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:26.549026 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3c986b-530c-4205-8180-dd32b661530d-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:42:27.206134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:27.206098 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" event={"ID":"cc3c986b-530c-4205-8180-dd32b661530d","Type":"ContainerDied","Data":"2bef53b570d12522df2bf0d2f08a147e84c1018bb9b854e8536c225efb8bbd4a"} Oct 31 18:42:27.206332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:27.206112 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf" Oct 31 18:42:27.206332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:42:27.206207 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bef53b570d12522df2bf0d2f08a147e84c1018bb9b854e8536c225efb8bbd4a" Oct 31 18:42:54.914608 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:42:54.914562 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:43:00.727479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.727450 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j"] Oct 31 18:43:00.727946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.727931 5688 cpu_manager.go:414] "containerMap: removing container" podUID="cc3c986b-530c-4205-8180-dd32b661530d" containerName="revoke-keys" Oct 31 18:43:00.727987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.727949 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3c986b-530c-4205-8180-dd32b661530d" containerName="revoke-keys" Oct 31 18:43:00.728051 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.728043 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3c986b-530c-4205-8180-dd32b661530d" containerName="revoke-keys" Oct 31 18:43:00.732802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.732785 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:00.836314 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.836284 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"distinct-w4-6279f5-kserve-self-signed-certs\"" Oct 31 18:43:00.839367 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.839344 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j"] Oct 31 18:43:00.901216 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.893612 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:00.901722 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.901627 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:00.901857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.901781 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:00.901857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.901836 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:00.902025 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.901913 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcw9\" (UniqueName: \"kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:00.902025 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:00.901989 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003405 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003373 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003422 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003472 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcw9\" (UniqueName: \"kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003524 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003744 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003604 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003744 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003641 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.003943 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003914 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.004076 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.003963 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.004148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.004125 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.006491 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.006464 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.006694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.006683 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.059784 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.059756 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcw9\" (UniqueName: \"kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9\") pod \"distinct-w4-6279f5-kserve-5944c6d77b-rfn6j\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.345030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.344941 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:01.744437 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.744408 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j"] Oct 31 18:43:01.748314 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:43:01.748289 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ca53f6_8a76_475a_8d2e_f87789d8cb9c.slice/crio-ac1c641cc5ffcca6a4199774cb117e9578155263790d71bbe1a0fcce8ba3a416 WatchSource:0}: Error finding container ac1c641cc5ffcca6a4199774cb117e9578155263790d71bbe1a0fcce8ba3a416: Status 404 returned error can't find the container with id ac1c641cc5ffcca6a4199774cb117e9578155263790d71bbe1a0fcce8ba3a416 Oct 31 18:43:01.757019 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:01.756996 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" event={"ID":"21ca53f6-8a76-475a-8d2e-f87789d8cb9c","Type":"ContainerStarted","Data":"ac1c641cc5ffcca6a4199774cb117e9578155263790d71bbe1a0fcce8ba3a416"} Oct 31 18:43:02.765171 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:02.765133 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" event={"ID":"21ca53f6-8a76-475a-8d2e-f87789d8cb9c","Type":"ContainerStarted","Data":"f61971f3521e53ed9f12cf4d544a68f035c753c39e4fec42a6b0cea1927673a0"} Oct 31 18:43:02.882655 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:02.882607 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" podStartSLOduration=2.882594768 podStartE2EDuration="2.882594768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:02.879027614 +0000 UTC m=+1303.481168758" watchObservedRunningTime="2026-10-31 18:43:02.882594768 +0000 UTC m=+1303.484735906" Oct 31 18:43:03.330602 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.330571 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6"] Oct 31 18:43:03.334521 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.334507 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.345499 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.345480 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:03.358280 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.358256 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.358381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.358311 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.358469 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.358451 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6w2\" (UniqueName: \"kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.389238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.388420 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:43:03.392243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.392216 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6"] Oct 31 18:43:03.437897 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.437861 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:43:03.459385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.459358 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.459494 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.459479 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6w2\" (UniqueName: \"kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.459611 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.459597 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.460164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.460134 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.461739 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.461714 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.516387 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.516362 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6w2\" (UniqueName: \"kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2\") pod \"maas-api-revoke-sub-67dbe3bb-24zx6\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:03.648043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:03.647981 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:04.022066 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:04.022038 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6"] Oct 31 18:43:04.023141 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:43:04.023118 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8500cd6b_c5d1_4a31_b1fa_908ad840cf35.slice/crio-5d23796a4da91b4106b2505f9b67814ecc906c50822d901fb22ddf6b5a18e43a WatchSource:0}: Error finding container 5d23796a4da91b4106b2505f9b67814ecc906c50822d901fb22ddf6b5a18e43a: Status 404 returned error can't find the container with id 5d23796a4da91b4106b2505f9b67814ecc906c50822d901fb22ddf6b5a18e43a Oct 31 18:43:04.779373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:04.779335 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" event={"ID":"8500cd6b-c5d1-4a31-b1fa-908ad840cf35","Type":"ContainerStarted","Data":"2b858a8a8c72ca9682889752b57bc5b6fdfbf2b6ec5cfcc8691460e1c39b23b5"} Oct 31 18:43:04.779556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:04.779389 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" event={"ID":"8500cd6b-c5d1-4a31-b1fa-908ad840cf35","Type":"ContainerStarted","Data":"5d23796a4da91b4106b2505f9b67814ecc906c50822d901fb22ddf6b5a18e43a"} Oct 31 18:43:04.892557 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:04.892396 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" podStartSLOduration=1.892381622 podStartE2EDuration="1.892381622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:04.891034459 +0000 UTC m=+1305.493175614" watchObservedRunningTime="2026-10-31 18:43:04.892381622 +0000 UTC m=+1305.494522767" Oct 31 18:43:05.788085 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:05.788047 5688 generic.go:356] "Generic (PLEG): container finished" podID="8500cd6b-c5d1-4a31-b1fa-908ad840cf35" containerID="2b858a8a8c72ca9682889752b57bc5b6fdfbf2b6ec5cfcc8691460e1c39b23b5" exitCode=0 Oct 31 18:43:05.788510 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:05.788195 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" event={"ID":"8500cd6b-c5d1-4a31-b1fa-908ad840cf35","Type":"ContainerDied","Data":"2b858a8a8c72ca9682889752b57bc5b6fdfbf2b6ec5cfcc8691460e1c39b23b5"} Oct 31 18:43:06.933057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:06.933030 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:07.002299 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.002269 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2\" (UniqueName: \"kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2\") pod \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " Oct 31 18:43:07.002465 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.002450 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca\") pod \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " Oct 31 18:43:07.002531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.002514 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca\") pod \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\" (UID: \"8500cd6b-c5d1-4a31-b1fa-908ad840cf35\") " Oct 31 18:43:07.002795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.002763 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca" pod "8500cd6b-c5d1-4a31-b1fa-908ad840cf35" (UID: "8500cd6b-c5d1-4a31-b1fa-908ad840cf35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:43:07.002957 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.002943 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:43:07.004403 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.004378 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2" pod "8500cd6b-c5d1-4a31-b1fa-908ad840cf35" (UID: "8500cd6b-c5d1-4a31-b1fa-908ad840cf35"). InnerVolumeSpecName "kube-api-access-vc6w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:43:07.004538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.004524 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca" pod "8500cd6b-c5d1-4a31-b1fa-908ad840cf35" (UID: "8500cd6b-c5d1-4a31-b1fa-908ad840cf35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:43:07.103887 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.103817 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:43:07.103887 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.103846 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6w2\" (UniqueName: \"kubernetes.io/projected/8500cd6b-c5d1-4a31-b1fa-908ad840cf35-kube-api-access-vc6w2\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:43:07.802341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.802315 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" Oct 31 18:43:07.802512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.802342 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6" event={"ID":"8500cd6b-c5d1-4a31-b1fa-908ad840cf35","Type":"ContainerDied","Data":"5d23796a4da91b4106b2505f9b67814ecc906c50822d901fb22ddf6b5a18e43a"} Oct 31 18:43:07.802512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:07.802396 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d23796a4da91b4106b2505f9b67814ecc906c50822d901fb22ddf6b5a18e43a" Oct 31 18:43:11.345721 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:11.345687 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:11.358278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:11.358255 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:11.843555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:11.843526 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 18:43:15.238416 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.238383 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm"] Oct 31 18:43:15.238900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.238885 5688 cpu_manager.go:414] "containerMap: removing container" podUID="8500cd6b-c5d1-4a31-b1fa-908ad840cf35" containerName="revoke-keys" Oct 31 18:43:15.238942 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.238901 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8500cd6b-c5d1-4a31-b1fa-908ad840cf35" containerName="revoke-keys" Oct 31 18:43:15.239000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.238990 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="8500cd6b-c5d1-4a31-b1fa-908ad840cf35" containerName="revoke-keys" Oct 31 18:43:15.242539 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.242522 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.294509 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.294479 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a6c8da-openshift-default-dockercfg-c5dr8\"" Oct 31 18:43:15.328678 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328644 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.328841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328695 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.328841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328714 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.328922 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328850 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.328922 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328917 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5vj\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.329009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328956 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.329009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.328999 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.329088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.329081 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.329131 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.329120 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.349826 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.349797 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm"] Oct 31 18:43:15.438987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.438927 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457359 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457409 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457521 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5vj\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457585 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457609 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457677 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457755 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.457869 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.462238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.447398 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.471213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.463808 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.483555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.472088 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.483795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.472381 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.483927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.472625 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.509291 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.505913 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.509291 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.507855 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.515754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.515723 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5vj\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.528596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.528574 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token\") pod \"e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.555771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.555745 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:15.988388 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:43:15.988360 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279d78ee_f449_4b03_8ef6_2b90eb588733.slice/crio-32c8758d06b711a20fbe7d3f528b4708b2921946eb4ab2fb6960fcbbd4951eb5 WatchSource:0}: Error finding container 32c8758d06b711a20fbe7d3f528b4708b2921946eb4ab2fb6960fcbbd4951eb5: Status 404 returned error can't find the container with id 32c8758d06b711a20fbe7d3f528b4708b2921946eb4ab2fb6960fcbbd4951eb5 Oct 31 18:43:15.991546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.991515 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm"] Oct 31 18:43:15.991652 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.991518 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:43:15.991652 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.991632 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:43:15.991745 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:15.991679 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:43:16.878150 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:16.878115 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" event={"ID":"279d78ee-f449-4b03-8ef6-2b90eb588733","Type":"ContainerStarted","Data":"0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca"} Oct 31 18:43:16.878528 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:16.878172 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" event={"ID":"279d78ee-f449-4b03-8ef6-2b90eb588733","Type":"ContainerStarted","Data":"32c8758d06b711a20fbe7d3f528b4708b2921946eb4ab2fb6960fcbbd4951eb5"} Oct 31 18:43:17.044548 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:17.044495 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podStartSLOduration=2.044480352 podStartE2EDuration="2.044480352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:16.993046207 +0000 UTC m=+1317.595187351" watchObservedRunningTime="2026-10-31 18:43:17.044480352 +0000 UTC m=+1317.646621565" Oct 31 18:43:17.556421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:17.556390 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:17.573299 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:17.573272 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:17.884206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:17.884099 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:17.885011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:17.884991 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:43:19.758682 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:19.758654 5688 scope.go:122] "RemoveContainer" containerID="4c5c5ff19f94a0037e6ef6cd0dd6b59a6f45c9b7579b2eced736b844683e922a" Oct 31 18:43:19.768167 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:19.768148 5688 scope.go:122] "RemoveContainer" containerID="4216dad6d281377bd99347c96381d37d9ef2ded92cc25b5bf346d8596532b7e9" Oct 31 18:43:19.776734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:19.776714 5688 scope.go:122] "RemoveContainer" containerID="484617f66d33932e8aff067e91c4f8788dfbfa4daa6439674518d184edae9181" Oct 31 18:43:19.788051 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:19.788029 5688 scope.go:122] "RemoveContainer" containerID="2d295beafc936b6fb2dfb2a18d566286f2e7ea70d94218660d3f74c3255fcaeb" Oct 31 18:43:24.904302 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:43:24.904268 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:43:54.941730 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:43:54.941654 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:43:57.362481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.362450 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq"] Oct 31 18:43:57.366868 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.366846 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.425557 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.425528 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-373109-openshift-default-dockercfg-q5g96\"" Oct 31 18:43:57.520960 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.520908 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.521317 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.521291 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.521516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.521501 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.521673 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.521652 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.521812 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.521798 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.521974 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.521957 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rb4\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.522201 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.522166 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.522377 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.522345 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.522516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.522502 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.660850 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.660920 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.660965 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.660990 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rb4\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.661126 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.661169 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.661256 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.661318 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.661347 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.662412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.662160 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.665082 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.665056 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.666531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.665471 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.667196 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.667159 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.667579 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.667560 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.681819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.681788 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq"] Oct 31 18:43:57.727246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.703479 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.742211 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.705085 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.773204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.772785 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.774722 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.774671 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rb4\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4\") pod \"e2e-api-b-373109-openshift-default-844b974c49-88jqq\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:57.980991 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:57.980869 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:43:58.417431 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:58.417396 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq"] Oct 31 18:43:58.418430 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:43:58.418402 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0da9986_1e4d_4af6_8b60_dc9ee793d4ba.slice/crio-1ef89a2b2af7ecb0192f7c413936bbd4c276d1c426b8ea4bd686000cc2923d84 WatchSource:0}: Error finding container 1ef89a2b2af7ecb0192f7c413936bbd4c276d1c426b8ea4bd686000cc2923d84: Status 404 returned error can't find the container with id 1ef89a2b2af7ecb0192f7c413936bbd4c276d1c426b8ea4bd686000cc2923d84 Oct 31 18:43:58.421028 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:58.420990 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:43:58.421133 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:58.421073 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:43:58.421133 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:58.421120 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:43:59.203537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.203498 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" event={"ID":"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba","Type":"ContainerStarted","Data":"5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292"} Oct 31 18:43:59.203818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.203800 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" event={"ID":"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba","Type":"ContainerStarted","Data":"1ef89a2b2af7ecb0192f7c413936bbd4c276d1c426b8ea4bd686000cc2923d84"} Oct 31 18:43:59.391070 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.391021 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podStartSLOduration=2.391007538 podStartE2EDuration="2.391007538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:59.387826324 +0000 UTC m=+1359.989967469" watchObservedRunningTime="2026-10-31 18:43:59.391007538 +0000 UTC m=+1359.993148736" Oct 31 18:43:59.709018 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.708988 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r"] Oct 31 18:43:59.712898 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.712879 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.766234 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.766152 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a6c8da-serving-cert\"" Oct 31 18:43:59.770807 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.770782 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r"] Oct 31 18:43:59.799717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.799688 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4z98\" (UniqueName: \"kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.799880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.799739 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.799880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.799859 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.800011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.799920 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.901099 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.901073 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4z98\" (UniqueName: \"kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.901296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.901106 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.901296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.901142 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.901296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.901175 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.901478 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:43:59.901315 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-serving-cert: secret "maas-api-e2e-api-a-a6c8da-serving-cert" not found Oct 31 18:43:59.901478 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:43:59.901367 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls podName:183949f0-d9c6-4980-9582-01b09818711d nodeName:}" failed. No retries permitted until 2026-10-31 18:44:00.401351212 +0000 UTC m=+1361.003492347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls") pod "maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" (UID: "183949f0-d9c6-4980-9582-01b09818711d") : secret "maas-api-e2e-api-a-a6c8da-serving-cert" not found Oct 31 18:43:59.901841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.901818 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.903571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.903551 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:43:59.991205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:43:59.987493 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:44:00.001852 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.001821 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4z98\" (UniqueName: \"kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:44:00.019555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.019517 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:44:00.231016 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.230986 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:44:00.232579 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.232555 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:44:00.407897 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.407822 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:44:00.410492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.410466 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\") pod \"maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:44:00.627309 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:00.627279 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:44:01.004018 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:01.003993 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r"] Oct 31 18:44:01.004519 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:44:01.004497 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183949f0_d9c6_4980_9582_01b09818711d.slice/crio-1a6290b0370602272147103daaa71f3c240be1ff0ebef6cf5e7ba78fd66d56b6 WatchSource:0}: Error finding container 1a6290b0370602272147103daaa71f3c240be1ff0ebef6cf5e7ba78fd66d56b6: Status 404 returned error can't find the container with id 1a6290b0370602272147103daaa71f3c240be1ff0ebef6cf5e7ba78fd66d56b6 Oct 31 18:44:01.005927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:01.005909 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 18:44:01.237011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:01.236974 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" event={"ID":"183949f0-d9c6-4980-9582-01b09818711d","Type":"ContainerStarted","Data":"1a6290b0370602272147103daaa71f3c240be1ff0ebef6cf5e7ba78fd66d56b6"} Oct 31 18:44:02.245648 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:02.245617 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" event={"ID":"183949f0-d9c6-4980-9582-01b09818711d","Type":"ContainerStarted","Data":"343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea"} Oct 31 18:44:02.246041 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:02.245697 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:44:02.364953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:02.364906 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" podStartSLOduration=2.488173265 podStartE2EDuration="3.364891486s" totalImagesPullingTime="876.718221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:44:02.362223789 +0000 UTC m=+1362.964364926" watchObservedRunningTime="2026-10-31 18:44:02.364891486 +0000 UTC m=+1362.967032624" Oct 31 18:44:06.162417 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.162371 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm"] Oct 31 18:44:06.172442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.172416 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.214010 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.213976 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm"] Oct 31 18:44:06.254294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.254267 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-373109\"" Oct 31 18:44:06.254294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.254267 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-373109-dockercfg-9xcc8\"" Oct 31 18:44:06.261898 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.261876 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.262053 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.262035 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jln\" (UniqueName: \"kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln\") pod \"payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.363646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.363621 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jln\" (UniqueName: \"kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln\") pod \"payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.363781 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.363727 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.364433 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.364415 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.419942 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.419886 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jln\" (UniqueName: \"kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln\") pod \"payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.493838 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.493813 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:06.870088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:06.870058 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm"] Oct 31 18:44:06.871667 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:44:06.871635 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1226be53_901c_499a_9e15_3dd6d0c29a6f.slice/crio-3bca1d24300cbef58b72b2825e9bf0323b2cc27350a99ead1f0920a182fe3fc0 WatchSource:0}: Error finding container 3bca1d24300cbef58b72b2825e9bf0323b2cc27350a99ead1f0920a182fe3fc0: Status 404 returned error can't find the container with id 3bca1d24300cbef58b72b2825e9bf0323b2cc27350a99ead1f0920a182fe3fc0 Oct 31 18:44:07.278374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:07.278343 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" event={"ID":"1226be53-901c-499a-9e15-3dd6d0c29a6f","Type":"ContainerStarted","Data":"3bca1d24300cbef58b72b2825e9bf0323b2cc27350a99ead1f0920a182fe3fc0"} Oct 31 18:44:08.258247 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:08.258218 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:44:10.303087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:10.303053 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" event={"ID":"1226be53-901c-499a-9e15-3dd6d0c29a6f","Type":"ContainerStarted","Data":"5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d"} Oct 31 18:44:10.303522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:10.303128 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:10.431207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:10.425416 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" podStartSLOduration=1.663085543 podStartE2EDuration="4.425398728s" totalImagesPullingTime="2.762313185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:44:10.424578009 +0000 UTC m=+1371.026719173" watchObservedRunningTime="2026-10-31 18:44:10.425398728 +0000 UTC m=+1371.027539873" Oct 31 18:44:21.313671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:21.313640 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:44:24.906293 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:44:24.906250 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:44:26.458281 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:26.458167 5688 generic.go:356] "Generic (PLEG): container finished" podID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerID="0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca" exitCode=0 Oct 31 18:44:26.458281 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:26.458221 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" event={"ID":"279d78ee-f449-4b03-8ef6-2b90eb588733","Type":"ContainerDied","Data":"0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca"} Oct 31 18:44:26.458732 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:26.458564 5688 scope.go:122] "RemoveContainer" containerID="0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca" Oct 31 18:44:26.459926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:26.459892 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:44:26.460047 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:26.459979 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:44:26.460047 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:26.460028 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:44:27.467370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:27.467329 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" event={"ID":"279d78ee-f449-4b03-8ef6-2b90eb588733","Type":"ContainerStarted","Data":"b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb"} Oct 31 18:44:27.557488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:27.557447 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:44:27.557704 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:27.557669 5688 patch_prober.go:29] interesting pod/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 18:44:27.557787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:27.557732 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 18:44:28.556587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:28.556551 5688 patch_prober.go:29] interesting pod/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 18:44:28.557206 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:28.557154 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 18:44:29.557603 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:29.557559 5688 patch_prober.go:29] interesting pod/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 18:44:29.558001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:29.557641 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 18:44:30.557597 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:30.557561 5688 patch_prober.go:29] interesting pod/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 18:44:30.557760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:30.557628 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 18:44:31.557555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:31.557517 5688 patch_prober.go:29] interesting pod/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 18:44:31.557760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:31.557587 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 18:44:32.560809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:32.560777 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:44:32.561245 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:32.561032 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:44:32.561792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:32.561778 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:44:40.053159 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:40.053126 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp"] Oct 31 18:44:40.074129 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:40.074095 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b7c162-gdnlp"] Oct 31 18:44:41.566601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:41.566570 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4661d39-dccb-417c-b579-eb984ecf82f9" path="/var/lib/kubelet/pods/a4661d39-dccb-417c-b579-eb984ecf82f9/volumes" Oct 31 18:44:47.674636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.674602 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk"] Oct 31 18:44:47.679023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.679002 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.735945 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.735925 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-373109-serving-cert\"" Oct 31 18:44:47.787701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.787675 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.787840 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.787706 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.787913 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.787889 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.787960 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.787943 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sc59\" (UniqueName: \"kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.789704 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.789676 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk"] Oct 31 18:44:47.889549 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.889517 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.889712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.889554 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.889712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.889651 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.889712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.889678 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sc59\" (UniqueName: \"kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.889820 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:44:47.889805 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-373109-serving-cert: secret "maas-api-e2e-api-b-373109-serving-cert" not found Oct 31 18:44:47.889886 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:44:47.889875 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls podName:0f49ffa2-202c-4dad-8608-8223ea8ff4e1 nodeName:}" failed. No retries permitted until 2026-10-31 18:44:48.389857811 +0000 UTC m=+1408.991998936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls") pod "maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" (UID: "0f49ffa2-202c-4dad-8608-8223ea8ff4e1") : secret "maas-api-e2e-api-b-373109-serving-cert" not found Oct 31 18:44:47.890277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.890259 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.892064 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.892048 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:47.993042 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:47.993007 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sc59\" (UniqueName: \"kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:48.397139 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:48.397054 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:48.399696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:48.399664 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\") pod \"maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:48.592215 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:48.592150 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:48.968577 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:48.968553 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk"] Oct 31 18:44:48.969842 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:44:48.969805 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f49ffa2_202c_4dad_8608_8223ea8ff4e1.slice/crio-7042bd83dd24fbb223d69c22f10d0ddf7b6536c581191b3cc736bf12bde3c55b WatchSource:0}: Error finding container 7042bd83dd24fbb223d69c22f10d0ddf7b6536c581191b3cc736bf12bde3c55b: Status 404 returned error can't find the container with id 7042bd83dd24fbb223d69c22f10d0ddf7b6536c581191b3cc736bf12bde3c55b Oct 31 18:44:49.640250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:49.640213 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" event={"ID":"0f49ffa2-202c-4dad-8608-8223ea8ff4e1","Type":"ContainerStarted","Data":"7042bd83dd24fbb223d69c22f10d0ddf7b6536c581191b3cc736bf12bde3c55b"} Oct 31 18:44:50.647370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:50.647335 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" event={"ID":"0f49ffa2-202c-4dad-8608-8223ea8ff4e1","Type":"ContainerStarted","Data":"c0287e0e43d6f90bdc699fd5c702114617ab02b4a6287cf2f44514eda2e02175"} Oct 31 18:44:50.647792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:50.647505 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:44:50.763311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:50.763264 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" podStartSLOduration=2.827258274 podStartE2EDuration="3.763251097s" totalImagesPullingTime="935.992823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:44:50.76089522 +0000 UTC m=+1411.363036365" watchObservedRunningTime="2026-10-31 18:44:50.763251097 +0000 UTC m=+1411.365392241" Oct 31 18:44:52.649709 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.649675 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5"] Oct 31 18:44:52.653989 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.653968 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.711154 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.711122 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cd7d17-openshift-default-dockercfg-rnx4k\"" Oct 31 18:44:52.756109 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.756069 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.756403 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.756380 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.767817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.767780 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.768029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.768010 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.768153 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.768136 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mh76\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.780749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.780707 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.796371 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.796338 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.813397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.813359 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.821587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.821552 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.868501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.868271 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5"] Oct 31 18:44:52.930357 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930250 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930357 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930306 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930357 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930341 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mh76\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930391 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930420 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930507 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930562 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930635 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.930903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.930670 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.932134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.931814 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.932134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.932104 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.932355 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.932273 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.932420 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.932403 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.932468 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.932417 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.934363 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.934345 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.956568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.956538 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.987688 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.987665 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:52.988474 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:52.988451 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mh76\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76\") pod \"e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:53.272646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:53.272609 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:53.760895 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:44:53.760858 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9298d17_2ffe_45c3_b494_4cbfa50e98e9.slice/crio-26396e958ee9ecda00616075f73a898dcd19e5f650a1ecd3981cce5a463f604a WatchSource:0}: Error finding container 26396e958ee9ecda00616075f73a898dcd19e5f650a1ecd3981cce5a463f604a: Status 404 returned error can't find the container with id 26396e958ee9ecda00616075f73a898dcd19e5f650a1ecd3981cce5a463f604a Oct 31 18:44:53.761753 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:53.761725 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5"] Oct 31 18:44:53.763463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:53.763423 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:44:53.763526 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:53.763507 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:44:53.763567 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:53.763545 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:44:54.049414 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:54.049309 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn"] Oct 31 18:44:54.052245 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:54.052220 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0f4379-q2rwn"] Oct 31 18:44:54.686657 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:54.686626 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" event={"ID":"f9298d17-2ffe-45c3-b494-4cbfa50e98e9","Type":"ContainerStarted","Data":"6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e"} Oct 31 18:44:54.686814 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:54.686678 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" event={"ID":"f9298d17-2ffe-45c3-b494-4cbfa50e98e9","Type":"ContainerStarted","Data":"26396e958ee9ecda00616075f73a898dcd19e5f650a1ecd3981cce5a463f604a"} Oct 31 18:44:54.854264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:54.854215 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podStartSLOduration=2.854201951 podStartE2EDuration="2.854201951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:44:54.85166603 +0000 UTC m=+1415.453807175" watchObservedRunningTime="2026-10-31 18:44:54.854201951 +0000 UTC m=+1415.456343091" Oct 31 18:44:54.904594 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:44:54.904561 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:44:55.272988 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:55.272955 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:55.278747 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:55.278726 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:55.567369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:55.567281 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1340650-82c6-45f7-9b08-7578575eab13" path="/var/lib/kubelet/pods/c1340650-82c6-45f7-9b08-7578575eab13/volumes" Oct 31 18:44:55.693036 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:55.693001 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:55.693892 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:55.693867 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:44:56.658437 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:44:56.658408 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:45:00.164756 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.164724 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4"] Oct 31 18:45:00.170688 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.170670 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t"] Oct 31 18:45:00.170807 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.170791 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.174727 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.174709 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.186980 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.186961 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk"] Oct 31 18:45:00.190951 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.190936 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.214111 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.214085 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5"] Oct 31 18:45:00.218230 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.218215 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.222624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.222603 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:45:00.269121 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.269095 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4"] Oct 31 18:45:00.273794 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.273766 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t"] Oct 31 18:45:00.315817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.315782 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5"] Oct 31 18:45:00.320662 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.320634 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk"] Oct 31 18:45:00.321646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.321617 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.321842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.321678 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8tv\" (UniqueName: \"kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.321842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.321803 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbml\" (UniqueName: \"kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml\") pod \"maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.321842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.321835 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.322020 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.321892 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtl62\" (UniqueName: \"kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62\") pod \"maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.322020 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.321966 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dc9l\" (UniqueName: \"kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l\") pod \"maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.322020 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.322004 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.322162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.322095 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.422971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.422898 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.422971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.422940 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8tv\" (UniqueName: \"kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.423175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.422974 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbml\" (UniqueName: \"kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml\") pod \"maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.423175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.422994 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.423175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423027 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtl62\" (UniqueName: \"kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62\") pod \"maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.423175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423080 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dc9l\" (UniqueName: \"kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l\") pod \"maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.423175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423111 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.423444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423229 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.423751 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423728 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.423787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423733 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.423903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423888 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.423939 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.423912 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.480649 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.480621 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dc9l\" (UniqueName: \"kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l\") pod \"maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.480807 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.480787 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbml\" (UniqueName: \"kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml\") pod \"maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.480882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.480789 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtl62\" (UniqueName: \"kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62\") pod \"maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.480882 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.480831 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8tv\" (UniqueName: \"kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.486478 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.486460 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:00.492397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.492378 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:00.506258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.506232 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:00.532059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.532035 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:00.888378 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.888348 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4"] Oct 31 18:45:00.889501 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:45:00.889474 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f562533_25ae_4a43_9533_42e07736bb3b.slice/crio-704747c45265cedb2554612413baa419b6fb84dbd640bb8fb69e0e2c9853590b WatchSource:0}: Error finding container 704747c45265cedb2554612413baa419b6fb84dbd640bb8fb69e0e2c9853590b: Status 404 returned error can't find the container with id 704747c45265cedb2554612413baa419b6fb84dbd640bb8fb69e0e2c9853590b Oct 31 18:45:00.938498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.938475 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t"] Oct 31 18:45:00.939238 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:45:00.939213 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060fd379_9d0c_4c89_a308_96d7e28c4dba.slice/crio-426bf63a05bd8c8d9793ec6755de8615162a300e032974f1611171b80b9f44d4 WatchSource:0}: Error finding container 426bf63a05bd8c8d9793ec6755de8615162a300e032974f1611171b80b9f44d4: Status 404 returned error can't find the container with id 426bf63a05bd8c8d9793ec6755de8615162a300e032974f1611171b80b9f44d4 Oct 31 18:45:00.939946 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:45:00.939923 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7169eee6_32d6_4588_abf6_11a270e7eedb.slice/crio-751bc75c9f2b72dcf3d755e9b8ff43d66f902f1d798024d6e2b5934e7ef44345 WatchSource:0}: Error finding container 751bc75c9f2b72dcf3d755e9b8ff43d66f902f1d798024d6e2b5934e7ef44345: Status 404 returned error can't find the container with id 751bc75c9f2b72dcf3d755e9b8ff43d66f902f1d798024d6e2b5934e7ef44345 Oct 31 18:45:00.941813 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.941782 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk"] Oct 31 18:45:00.989543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:00.989524 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5"] Oct 31 18:45:00.990517 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:45:00.990495 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc77896_1923_4dc0_b6d8_ded5a444e8d8.slice/crio-43b141461b36664b94d8268e554cb8e2b4e67f651253c9a06fcd874ca9fac061 WatchSource:0}: Error finding container 43b141461b36664b94d8268e554cb8e2b4e67f651253c9a06fcd874ca9fac061: Status 404 returned error can't find the container with id 43b141461b36664b94d8268e554cb8e2b4e67f651253c9a06fcd874ca9fac061 Oct 31 18:45:01.738890 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.738857 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerStarted","Data":"461e93e21a7b93af731ac8160aa198ef7c7163b10296b39fdc1520c56bbf22ab"} Oct 31 18:45:01.739336 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.738915 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerStarted","Data":"426bf63a05bd8c8d9793ec6755de8615162a300e032974f1611171b80b9f44d4"} Oct 31 18:45:01.740625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.740600 5688 generic.go:356] "Generic (PLEG): container finished" podID="5f562533-25ae-4a43-9533-42e07736bb3b" containerID="f97c48c0a536c0e0511dbb3bbd9945ced6d2ea1f315684d8c4698ec2db2b3c1a" exitCode=60 Oct 31 18:45:01.740717 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.740668 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" event={"ID":"5f562533-25ae-4a43-9533-42e07736bb3b","Type":"ContainerDied","Data":"f97c48c0a536c0e0511dbb3bbd9945ced6d2ea1f315684d8c4698ec2db2b3c1a"} Oct 31 18:45:01.740773 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.740711 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" event={"ID":"5f562533-25ae-4a43-9533-42e07736bb3b","Type":"ContainerStarted","Data":"704747c45265cedb2554612413baa419b6fb84dbd640bb8fb69e0e2c9853590b"} Oct 31 18:45:01.740857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.740842 5688 scope.go:122] "RemoveContainer" containerID="f97c48c0a536c0e0511dbb3bbd9945ced6d2ea1f315684d8c4698ec2db2b3c1a" Oct 31 18:45:01.742389 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.742363 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerStarted","Data":"0b0ef085345c3395ef90444126dc08124d87c489d2bc6cc75c72c9b650b2df4a"} Oct 31 18:45:01.742493 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.742415 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerStarted","Data":"751bc75c9f2b72dcf3d755e9b8ff43d66f902f1d798024d6e2b5934e7ef44345"} Oct 31 18:45:01.744435 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.744291 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerStarted","Data":"1d2f602b86538f30e477120cb1c5a8b0f1ab581f7e7fd6b0c7a9cdc97ab24dce"} Oct 31 18:45:01.744435 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.744336 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerStarted","Data":"43b141461b36664b94d8268e554cb8e2b4e67f651253c9a06fcd874ca9fac061"} Oct 31 18:45:01.853948 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.853903 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" podStartSLOduration=1.8538887229999998 podStartE2EDuration="1.853888723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:01.850623523 +0000 UTC m=+1422.452764667" watchObservedRunningTime="2026-10-31 18:45:01.853888723 +0000 UTC m=+1422.456029877" Oct 31 18:45:01.964285 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:01.964232 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" podStartSLOduration=1.9642167769999999 podStartE2EDuration="1.964216777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:01.962381088 +0000 UTC m=+1422.564522231" watchObservedRunningTime="2026-10-31 18:45:01.964216777 +0000 UTC m=+1422.566357932" Oct 31 18:45:02.073829 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.073788 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" podStartSLOduration=2.073775093 podStartE2EDuration="2.073775093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:02.07366524 +0000 UTC m=+1422.675806380" watchObservedRunningTime="2026-10-31 18:45:02.073775093 +0000 UTC m=+1422.675916236" Oct 31 18:45:02.755061 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.755030 5688 generic.go:356] "Generic (PLEG): container finished" podID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerID="1d2f602b86538f30e477120cb1c5a8b0f1ab581f7e7fd6b0c7a9cdc97ab24dce" exitCode=60 Oct 31 18:45:02.755535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.755122 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerDied","Data":"1d2f602b86538f30e477120cb1c5a8b0f1ab581f7e7fd6b0c7a9cdc97ab24dce"} Oct 31 18:45:02.755600 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.755563 5688 scope.go:122] "RemoveContainer" containerID="1d2f602b86538f30e477120cb1c5a8b0f1ab581f7e7fd6b0c7a9cdc97ab24dce" Oct 31 18:45:02.757479 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.757426 5688 generic.go:356] "Generic (PLEG): container finished" podID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerID="461e93e21a7b93af731ac8160aa198ef7c7163b10296b39fdc1520c56bbf22ab" exitCode=60 Oct 31 18:45:02.757548 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.757510 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerDied","Data":"461e93e21a7b93af731ac8160aa198ef7c7163b10296b39fdc1520c56bbf22ab"} Oct 31 18:45:02.758053 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.757801 5688 scope.go:122] "RemoveContainer" containerID="461e93e21a7b93af731ac8160aa198ef7c7163b10296b39fdc1520c56bbf22ab" Oct 31 18:45:02.761738 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.761716 5688 generic.go:356] "Generic (PLEG): container finished" podID="5f562533-25ae-4a43-9533-42e07736bb3b" containerID="b76b26fba511681fc776c3fa4ee03b873cc8a9a96de35cd0fd355ae48d433e86" exitCode=60 Oct 31 18:45:02.761844 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.761828 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" event={"ID":"5f562533-25ae-4a43-9533-42e07736bb3b","Type":"ContainerDied","Data":"b76b26fba511681fc776c3fa4ee03b873cc8a9a96de35cd0fd355ae48d433e86"} Oct 31 18:45:02.761905 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.761870 5688 scope.go:122] "RemoveContainer" containerID="f97c48c0a536c0e0511dbb3bbd9945ced6d2ea1f315684d8c4698ec2db2b3c1a" Oct 31 18:45:02.762569 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.762463 5688 scope.go:122] "RemoveContainer" containerID="b76b26fba511681fc776c3fa4ee03b873cc8a9a96de35cd0fd355ae48d433e86" Oct 31 18:45:02.762723 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:02.762700 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4_odh-ai-gateway-infra(5f562533-25ae-4a43-9533-42e07736bb3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" Oct 31 18:45:02.764571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.764550 5688 generic.go:356] "Generic (PLEG): container finished" podID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerID="0b0ef085345c3395ef90444126dc08124d87c489d2bc6cc75c72c9b650b2df4a" exitCode=60 Oct 31 18:45:02.764668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.764632 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerDied","Data":"0b0ef085345c3395ef90444126dc08124d87c489d2bc6cc75c72c9b650b2df4a"} Oct 31 18:45:02.764844 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:02.764827 5688 scope.go:122] "RemoveContainer" containerID="0b0ef085345c3395ef90444126dc08124d87c489d2bc6cc75c72c9b650b2df4a" Oct 31 18:45:03.772726 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.772697 5688 generic.go:356] "Generic (PLEG): container finished" podID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerID="42e89fbf6775ed5ca70aabc80d96abd19bbc2fdb22a09f8300a1d3458b1a2f6a" exitCode=60 Oct 31 18:45:03.773084 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.772771 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerDied","Data":"42e89fbf6775ed5ca70aabc80d96abd19bbc2fdb22a09f8300a1d3458b1a2f6a"} Oct 31 18:45:03.773084 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.772824 5688 scope.go:122] "RemoveContainer" containerID="461e93e21a7b93af731ac8160aa198ef7c7163b10296b39fdc1520c56bbf22ab" Oct 31 18:45:03.773084 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.773050 5688 scope.go:122] "RemoveContainer" containerID="42e89fbf6775ed5ca70aabc80d96abd19bbc2fdb22a09f8300a1d3458b1a2f6a" Oct 31 18:45:03.773396 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:03.773376 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t_odh-ai-gateway-infra(060fd379-9d0c-4c89-a308-96d7e28c4dba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" Oct 31 18:45:03.774828 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.774812 5688 scope.go:122] "RemoveContainer" containerID="b76b26fba511681fc776c3fa4ee03b873cc8a9a96de35cd0fd355ae48d433e86" Oct 31 18:45:03.775085 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:03.775064 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4_odh-ai-gateway-infra(5f562533-25ae-4a43-9533-42e07736bb3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" Oct 31 18:45:03.776405 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.776383 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerStarted","Data":"d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16"} Oct 31 18:45:03.778729 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.778703 5688 generic.go:356] "Generic (PLEG): container finished" podID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerID="d62d7df8e6db2314d11e55215b75d4751b5dbae6a56f6f6ccedaf097164e3bce" exitCode=60 Oct 31 18:45:03.778916 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.778754 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerDied","Data":"d62d7df8e6db2314d11e55215b75d4751b5dbae6a56f6f6ccedaf097164e3bce"} Oct 31 18:45:03.779410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.779070 5688 scope.go:122] "RemoveContainer" containerID="d62d7df8e6db2314d11e55215b75d4751b5dbae6a56f6f6ccedaf097164e3bce" Oct 31 18:45:03.779410 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:03.779355 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5_odh-ai-gateway-infra(7bc77896-1923-4dc0-b6d8-ded5a444e8d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" Oct 31 18:45:03.878142 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:03.878123 5688 scope.go:122] "RemoveContainer" containerID="1d2f602b86538f30e477120cb1c5a8b0f1ab581f7e7fd6b0c7a9cdc97ab24dce" Oct 31 18:45:04.788301 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:04.788273 5688 scope.go:122] "RemoveContainer" containerID="d62d7df8e6db2314d11e55215b75d4751b5dbae6a56f6f6ccedaf097164e3bce" Oct 31 18:45:04.788777 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:04.788626 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5_odh-ai-gateway-infra(7bc77896-1923-4dc0-b6d8-ded5a444e8d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" Oct 31 18:45:04.789938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:04.789915 5688 scope.go:122] "RemoveContainer" containerID="42e89fbf6775ed5ca70aabc80d96abd19bbc2fdb22a09f8300a1d3458b1a2f6a" Oct 31 18:45:04.790259 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:04.790231 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t_odh-ai-gateway-infra(060fd379-9d0c-4c89-a308-96d7e28c4dba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" Oct 31 18:45:04.791530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:04.791509 5688 generic.go:356] "Generic (PLEG): container finished" podID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerID="d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16" exitCode=60 Oct 31 18:45:04.791592 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:04.791549 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerDied","Data":"d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16"} Oct 31 18:45:04.791592 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:04.791589 5688 scope.go:122] "RemoveContainer" containerID="0b0ef085345c3395ef90444126dc08124d87c489d2bc6cc75c72c9b650b2df4a" Oct 31 18:45:04.791785 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:04.791771 5688 scope.go:122] "RemoveContainer" containerID="d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16" Oct 31 18:45:04.792017 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:04.791997 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk_odh-ai-gateway-infra(7169eee6-32d6-4588-abf6-11a270e7eedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" Oct 31 18:45:05.799988 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:05.799962 5688 scope.go:122] "RemoveContainer" containerID="d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16" Oct 31 18:45:05.800439 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:05.800266 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk_odh-ai-gateway-infra(7169eee6-32d6-4588-abf6-11a270e7eedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" Oct 31 18:45:11.081212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:11.071974 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk"] Oct 31 18:45:11.098131 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:11.098006 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c837a9-qknhk"] Oct 31 18:45:11.615209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:11.605264 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7e1faa-342d-4954-8394-c4b33038f3eb" path="/var/lib/kubelet/pods/5f7e1faa-342d-4954-8394-c4b33038f3eb/volumes" Oct 31 18:45:13.569121 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:13.569091 5688 scope.go:122] "RemoveContainer" containerID="b76b26fba511681fc776c3fa4ee03b873cc8a9a96de35cd0fd355ae48d433e86" Oct 31 18:45:15.103213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:15.103155 5688 generic.go:356] "Generic (PLEG): container finished" podID="5f562533-25ae-4a43-9533-42e07736bb3b" containerID="9403a7f008625c74ecc06b7f59ba3bad1270b3e3f33dbd762bdc483453a15f4e" exitCode=60 Oct 31 18:45:15.103585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:15.103235 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" event={"ID":"5f562533-25ae-4a43-9533-42e07736bb3b","Type":"ContainerDied","Data":"9403a7f008625c74ecc06b7f59ba3bad1270b3e3f33dbd762bdc483453a15f4e"} Oct 31 18:45:15.103585 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:15.103339 5688 scope.go:122] "RemoveContainer" containerID="b76b26fba511681fc776c3fa4ee03b873cc8a9a96de35cd0fd355ae48d433e86" Oct 31 18:45:15.103846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:15.103824 5688 scope.go:122] "RemoveContainer" containerID="9403a7f008625c74ecc06b7f59ba3bad1270b3e3f33dbd762bdc483453a15f4e" Oct 31 18:45:15.104228 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:15.104177 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4_odh-ai-gateway-infra(5f562533-25ae-4a43-9533-42e07736bb3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" Oct 31 18:45:15.560795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:15.560760 5688 scope.go:122] "RemoveContainer" containerID="d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16" Oct 31 18:45:16.113146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.113112 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerStarted","Data":"e8b306ed6e54b545d90673f1dcb993cc6b48cbb061020b6d1355abf621970c75"} Oct 31 18:45:16.278644 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.278609 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4"] Oct 31 18:45:16.504012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.503988 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:16.560334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.560291 5688 scope.go:122] "RemoveContainer" containerID="42e89fbf6775ed5ca70aabc80d96abd19bbc2fdb22a09f8300a1d3458b1a2f6a" Oct 31 18:45:16.697710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.697680 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca\") pod \"5f562533-25ae-4a43-9533-42e07736bb3b\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " Oct 31 18:45:16.697872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.697797 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml\" (UniqueName: \"kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml\") pod \"5f562533-25ae-4a43-9533-42e07736bb3b\" (UID: \"5f562533-25ae-4a43-9533-42e07736bb3b\") " Oct 31 18:45:16.698097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.698075 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca" pod "5f562533-25ae-4a43-9533-42e07736bb3b" (UID: "5f562533-25ae-4a43-9533-42e07736bb3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:45:16.698317 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.698302 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f562533-25ae-4a43-9533-42e07736bb3b-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:16.764481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.764436 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml" pod "5f562533-25ae-4a43-9533-42e07736bb3b" (UID: "5f562533-25ae-4a43-9533-42e07736bb3b"). InnerVolumeSpecName "kube-api-access-2bbml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:45:16.799262 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:16.799238 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bbml\" (UniqueName: \"kubernetes.io/projected/5f562533-25ae-4a43-9533-42e07736bb3b-kube-api-access-2bbml\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:17.122869 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.122838 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerStarted","Data":"3cfd153d48e79698705cabb722d670e282846422e73866cd7950bca1718a4aec"} Oct 31 18:45:17.124308 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.124289 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" Oct 31 18:45:17.124436 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.124323 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4" event={"ID":"5f562533-25ae-4a43-9533-42e07736bb3b","Type":"ContainerDied","Data":"704747c45265cedb2554612413baa419b6fb84dbd640bb8fb69e0e2c9853590b"} Oct 31 18:45:17.124436 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.124379 5688 scope.go:122] "RemoveContainer" containerID="9403a7f008625c74ecc06b7f59ba3bad1270b3e3f33dbd762bdc483453a15f4e" Oct 31 18:45:17.126388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.126368 5688 generic.go:356] "Generic (PLEG): container finished" podID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerID="e8b306ed6e54b545d90673f1dcb993cc6b48cbb061020b6d1355abf621970c75" exitCode=60 Oct 31 18:45:17.126483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.126430 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerDied","Data":"e8b306ed6e54b545d90673f1dcb993cc6b48cbb061020b6d1355abf621970c75"} Oct 31 18:45:17.126780 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.126765 5688 scope.go:122] "RemoveContainer" containerID="e8b306ed6e54b545d90673f1dcb993cc6b48cbb061020b6d1355abf621970c75" Oct 31 18:45:17.127042 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:17.127020 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk_odh-ai-gateway-infra(7169eee6-32d6-4588-abf6-11a270e7eedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" Oct 31 18:45:17.159550 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.159531 5688 scope.go:122] "RemoveContainer" containerID="d257843519d5dd38586ac6e333862238b6d771daed5a44ad6d2fa3fa6e9cbb16" Oct 31 18:45:17.268609 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.268576 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk"] Oct 31 18:45:17.367730 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.367694 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4"] Oct 31 18:45:17.379534 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.379501 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a6c8da-29891205-nhzs4"] Oct 31 18:45:17.559868 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.559844 5688 scope.go:122] "RemoveContainer" containerID="d62d7df8e6db2314d11e55215b75d4751b5dbae6a56f6f6ccedaf097164e3bce" Oct 31 18:45:17.570158 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:17.570130 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" path="/var/lib/kubelet/pods/5f562533-25ae-4a43-9533-42e07736bb3b/volumes" Oct 31 18:45:18.134636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.134554 5688 generic.go:356] "Generic (PLEG): container finished" podID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerID="3cfd153d48e79698705cabb722d670e282846422e73866cd7950bca1718a4aec" exitCode=60 Oct 31 18:45:18.135074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.134630 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerDied","Data":"3cfd153d48e79698705cabb722d670e282846422e73866cd7950bca1718a4aec"} Oct 31 18:45:18.135074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.134696 5688 scope.go:122] "RemoveContainer" containerID="42e89fbf6775ed5ca70aabc80d96abd19bbc2fdb22a09f8300a1d3458b1a2f6a" Oct 31 18:45:18.135234 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.135091 5688 scope.go:122] "RemoveContainer" containerID="3cfd153d48e79698705cabb722d670e282846422e73866cd7950bca1718a4aec" Oct 31 18:45:18.135444 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:18.135422 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t_odh-ai-gateway-infra(060fd379-9d0c-4c89-a308-96d7e28c4dba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" Oct 31 18:45:18.140043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.140021 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerStarted","Data":"1a487c9ee774bfb987dc45cc38f2d217fbba02431d3b8f00bfdfa1177a8876ee"} Oct 31 18:45:18.253409 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.253378 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t"] Oct 31 18:45:18.311289 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.311261 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:18.319156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.317571 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca\") pod \"7169eee6-32d6-4588-abf6-11a270e7eedb\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " Oct 31 18:45:18.319156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.317703 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l\" (UniqueName: \"kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l\") pod \"7169eee6-32d6-4588-abf6-11a270e7eedb\" (UID: \"7169eee6-32d6-4588-abf6-11a270e7eedb\") " Oct 31 18:45:18.319385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.319321 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca" pod "7169eee6-32d6-4588-abf6-11a270e7eedb" (UID: "7169eee6-32d6-4588-abf6-11a270e7eedb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:45:18.321165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.321139 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l" pod "7169eee6-32d6-4588-abf6-11a270e7eedb" (UID: "7169eee6-32d6-4588-abf6-11a270e7eedb"). InnerVolumeSpecName "kube-api-access-6dc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:45:18.420829 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.420709 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dc9l\" (UniqueName: \"kubernetes.io/projected/7169eee6-32d6-4588-abf6-11a270e7eedb-kube-api-access-6dc9l\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:18.420829 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:18.420741 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7169eee6-32d6-4588-abf6-11a270e7eedb-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:19.149919 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.149895 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" Oct 31 18:45:19.150303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.149920 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk" event={"ID":"7169eee6-32d6-4588-abf6-11a270e7eedb","Type":"ContainerDied","Data":"751bc75c9f2b72dcf3d755e9b8ff43d66f902f1d798024d6e2b5934e7ef44345"} Oct 31 18:45:19.150303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.149995 5688 scope.go:122] "RemoveContainer" containerID="e8b306ed6e54b545d90673f1dcb993cc6b48cbb061020b6d1355abf621970c75" Oct 31 18:45:19.152492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.152464 5688 generic.go:356] "Generic (PLEG): container finished" podID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerID="1a487c9ee774bfb987dc45cc38f2d217fbba02431d3b8f00bfdfa1177a8876ee" exitCode=60 Oct 31 18:45:19.152597 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.152554 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerDied","Data":"1a487c9ee774bfb987dc45cc38f2d217fbba02431d3b8f00bfdfa1177a8876ee"} Oct 31 18:45:19.152961 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.152943 5688 scope.go:122] "RemoveContainer" containerID="1a487c9ee774bfb987dc45cc38f2d217fbba02431d3b8f00bfdfa1177a8876ee" Oct 31 18:45:19.153295 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:19.153271 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5_odh-ai-gateway-infra(7bc77896-1923-4dc0-b6d8-ded5a444e8d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" Oct 31 18:45:19.161492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.161471 5688 scope.go:122] "RemoveContainer" containerID="d62d7df8e6db2314d11e55215b75d4751b5dbae6a56f6f6ccedaf097164e3bce" Oct 31 18:45:19.315285 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.315259 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:19.332393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.332368 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca\") pod \"060fd379-9d0c-4c89-a308-96d7e28c4dba\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " Oct 31 18:45:19.332610 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.332591 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv\" (UniqueName: \"kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv\") pod \"060fd379-9d0c-4c89-a308-96d7e28c4dba\" (UID: \"060fd379-9d0c-4c89-a308-96d7e28c4dba\") " Oct 31 18:45:19.332709 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.332684 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca" pod "060fd379-9d0c-4c89-a308-96d7e28c4dba" (UID: "060fd379-9d0c-4c89-a308-96d7e28c4dba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:45:19.333009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.332984 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd379-9d0c-4c89-a308-96d7e28c4dba-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:19.334946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.334922 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv" pod "060fd379-9d0c-4c89-a308-96d7e28c4dba" (UID: "060fd379-9d0c-4c89-a308-96d7e28c4dba"). InnerVolumeSpecName "kube-api-access-mk8tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:45:19.376506 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.376324 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk"] Oct 31 18:45:19.379686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.379663 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd6c0a-29891205-fqzxk"] Oct 31 18:45:19.433927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.433853 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8tv\" (UniqueName: \"kubernetes.io/projected/060fd379-9d0c-4c89-a308-96d7e28c4dba-kube-api-access-mk8tv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:19.523363 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.523335 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5"] Oct 31 18:45:19.568620 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:19.568590 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" path="/var/lib/kubelet/pods/7169eee6-32d6-4588-abf6-11a270e7eedb/volumes" Oct 31 18:45:20.164767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.164742 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" Oct 31 18:45:20.165128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.164762 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t" event={"ID":"060fd379-9d0c-4c89-a308-96d7e28c4dba","Type":"ContainerDied","Data":"426bf63a05bd8c8d9793ec6755de8615162a300e032974f1611171b80b9f44d4"} Oct 31 18:45:20.165128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.164868 5688 scope.go:122] "RemoveContainer" containerID="3cfd153d48e79698705cabb722d670e282846422e73866cd7950bca1718a4aec" Oct 31 18:45:20.327652 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.327631 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:20.346388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.346364 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca\") pod \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " Oct 31 18:45:20.346489 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.346399 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62\" (UniqueName: \"kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62\") pod \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\" (UID: \"7bc77896-1923-4dc0-b6d8-ded5a444e8d8\") " Oct 31 18:45:20.346741 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.346717 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca" pod "7bc77896-1923-4dc0-b6d8-ded5a444e8d8" (UID: "7bc77896-1923-4dc0-b6d8-ded5a444e8d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:45:20.346907 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.346891 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:20.348519 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.348497 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62" pod "7bc77896-1923-4dc0-b6d8-ded5a444e8d8" (UID: "7bc77896-1923-4dc0-b6d8-ded5a444e8d8"). InnerVolumeSpecName "kube-api-access-gtl62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:45:20.404905 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.404878 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t"] Oct 31 18:45:20.408421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.408400 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891205-cfk2t"] Oct 31 18:45:20.447828 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:20.447765 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtl62\" (UniqueName: \"kubernetes.io/projected/7bc77896-1923-4dc0-b6d8-ded5a444e8d8-kube-api-access-gtl62\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:45:21.173647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.173547 5688 generic.go:356] "Generic (PLEG): container finished" podID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerID="5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292" exitCode=0 Oct 31 18:45:21.173647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.173614 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" event={"ID":"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba","Type":"ContainerDied","Data":"5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292"} Oct 31 18:45:21.174157 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.174043 5688 scope.go:122] "RemoveContainer" containerID="5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292" Oct 31 18:45:21.175141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.175121 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" Oct 31 18:45:21.175264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.175137 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5" event={"ID":"7bc77896-1923-4dc0-b6d8-ded5a444e8d8","Type":"ContainerDied","Data":"43b141461b36664b94d8268e554cb8e2b4e67f651253c9a06fcd874ca9fac061"} Oct 31 18:45:21.175264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.175220 5688 scope.go:122] "RemoveContainer" containerID="1a487c9ee774bfb987dc45cc38f2d217fbba02431d3b8f00bfdfa1177a8876ee" Oct 31 18:45:21.175621 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.175547 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:45:21.175707 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.175634 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:45:21.175769 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.175709 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:45:21.492330 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.492299 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5"] Oct 31 18:45:21.496983 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.496956 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3417b822-29891205-sj7c5"] Oct 31 18:45:21.569582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.569552 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" path="/var/lib/kubelet/pods/060fd379-9d0c-4c89-a308-96d7e28c4dba/volumes" Oct 31 18:45:21.569975 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:21.569962 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" path="/var/lib/kubelet/pods/7bc77896-1923-4dc0-b6d8-ded5a444e8d8/volumes" Oct 31 18:45:22.017232 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.017197 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5"] Oct 31 18:45:22.018373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018346 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:22.018533 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018521 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:22.018633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018621 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:22.018713 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018703 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:22.018800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018791 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:22.018871 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018862 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:22.018955 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.018946 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.019022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.019013 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.019097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.019086 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.035268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035234 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.035501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035487 5688 cpu_manager.go:414] "containerMap: removing container" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:22.035594 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035583 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:22.035676 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035667 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:22.035748 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035737 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:22.035849 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035837 5688 cpu_manager.go:414] "containerMap: removing container" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:22.035931 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.035922 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:22.036227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036205 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.036338 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036328 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.036414 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036406 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:22.036484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036476 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:22.036554 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036546 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:22.036626 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036618 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:22.036699 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036691 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:22.036767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036758 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:22.036832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.036825 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:22.043213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.041537 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.052934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.052359 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5"] Oct 31 18:45:22.095207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.094491 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c21e9e-dockercfg-m2ksr\"" Oct 31 18:45:22.095207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.094578 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c21e9e\"" Oct 31 18:45:22.116098 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.112959 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ljv\" (UniqueName: \"kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv\") pod \"payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.116098 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.113069 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.287267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.286768 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ljv\" (UniqueName: \"kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv\") pod \"payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.297240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.292315 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.316699 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.316665 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.331219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.321881 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" event={"ID":"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba","Type":"ContainerStarted","Data":"bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a"} Oct 31 18:45:22.402248 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.402216 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ljv\" (UniqueName: \"kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv\") pod \"payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.686915 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.686828 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:22.981892 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.981822 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:45:22.982059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.982027 5688 patch_prober.go:29] interesting pod/e2e-api-b-373109-openshift-default-844b974c49-88jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 18:45:22.982139 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:22.982077 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 18:45:23.091860 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.091835 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5"] Oct 31 18:45:23.093363 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:45:23.093336 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28aa4a8_8829_4a3c_ae7e_f6e1b190813b.slice/crio-bf032514dd48554f2d1ac7b06330adb4111183d1faaf62f2a51450659e254f35 WatchSource:0}: Error finding container bf032514dd48554f2d1ac7b06330adb4111183d1faaf62f2a51450659e254f35: Status 404 returned error can't find the container with id bf032514dd48554f2d1ac7b06330adb4111183d1faaf62f2a51450659e254f35 Oct 31 18:45:23.348210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.345088 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" event={"ID":"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b","Type":"ContainerStarted","Data":"0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f"} Oct 31 18:45:23.348210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.345146 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" event={"ID":"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b","Type":"ContainerStarted","Data":"bf032514dd48554f2d1ac7b06330adb4111183d1faaf62f2a51450659e254f35"} Oct 31 18:45:23.348210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.345216 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:23.460397 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.460336 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" podStartSLOduration=2.460319581 podStartE2EDuration="2.460319581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:23.459756761 +0000 UTC m=+1444.061897905" watchObservedRunningTime="2026-10-31 18:45:23.460319581 +0000 UTC m=+1444.062460725" Oct 31 18:45:23.982064 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.982032 5688 patch_prober.go:29] interesting pod/e2e-api-b-373109-openshift-default-844b974c49-88jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 18:45:23.982259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:23.982081 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 18:45:24.904624 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:24.904589 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:45:24.981743 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:24.981712 5688 patch_prober.go:29] interesting pod/e2e-api-b-373109-openshift-default-844b974c49-88jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 18:45:24.981908 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:24.981771 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 18:45:25.982103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:25.982071 5688 patch_prober.go:29] interesting pod/e2e-api-b-373109-openshift-default-844b974c49-88jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 18:45:25.982508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:25.982124 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 18:45:26.981342 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:26.981310 5688 patch_prober.go:29] interesting pod/e2e-api-b-373109-openshift-default-844b974c49-88jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 18:45:26.981518 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:26.981359 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 18:45:27.058934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:27.058893 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6"] Oct 31 18:45:27.062154 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:27.062124 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-888b8e32-hvnf6"] Oct 31 18:45:27.568030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:27.567999 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3" path="/var/lib/kubelet/pods/b0228b6b-b1b2-4682-b1ab-7cd6ed7843f3/volumes" Oct 31 18:45:27.986862 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:27.986833 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:45:27.987488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:27.987462 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:45:27.988324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:27.988303 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:45:35.371281 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:35.371249 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:45:37.047369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:37.047333 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn"] Oct 31 18:45:37.096712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:37.096684 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d6538f-z4lgn"] Oct 31 18:45:37.566267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:37.566236 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0721eb41-5043-4c2b-9e03-89339507c4c4" path="/var/lib/kubelet/pods/0721eb41-5043-4c2b-9e03-89339507c4c4/volumes" Oct 31 18:45:54.904746 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:45:54.904711 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:45:56.045855 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:56.045826 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn"] Oct 31 18:45:56.094846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:56.094816 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6badabd3-5fbtn"] Oct 31 18:45:57.305983 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.305953 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl"] Oct 31 18:45:57.306587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306568 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:57.306587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306586 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:57.306710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306606 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:57.306710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306613 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f562533-25ae-4a43-9533-42e07736bb3b" containerName="cleanup" Oct 31 18:45:57.306710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306626 5688 cpu_manager.go:414] "containerMap: removing container" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:57.306710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306631 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:57.306710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306641 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:57.306710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306645 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:57.306971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306738 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7169eee6-32d6-4588-abf6-11a270e7eedb" containerName="cleanup" Oct 31 18:45:57.306971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306749 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc77896-1923-4dc0-b6d8-ded5a444e8d8" containerName="cleanup" Oct 31 18:45:57.306971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.306755 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="060fd379-9d0c-4c89-a308-96d7e28c4dba" containerName="cleanup" Oct 31 18:45:57.311422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.311401 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.366097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.366072 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl"] Oct 31 18:45:57.412241 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.412208 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:45:57.412372 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.412241 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:45:57.468060 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.468028 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.468227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.468071 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.468227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.468157 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bknn\" (UniqueName: \"kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.566927 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.566860 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254e50e9-df69-4ff7-8846-5ad6822f7723" path="/var/lib/kubelet/pods/254e50e9-df69-4ff7-8846-5ad6822f7723/volumes" Oct 31 18:45:57.569054 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.569025 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.569218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.569064 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.569218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.569106 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bknn\" (UniqueName: \"kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.569777 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.569751 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.571853 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.571831 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.628042 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.628015 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bknn\" (UniqueName: \"kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn\") pod \"maas-api-revoke-sub-56e11e61-smdcl\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:57.924164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:57.924090 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:45:58.301442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:58.301417 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl"] Oct 31 18:45:58.304050 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:45:58.304025 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b4306_1468_44ae_b183_66c9d9a8c330.slice/crio-12a0d82b88fed864e3bf32b7b41a4af3a37b1c828d858e0de26b002af392910a WatchSource:0}: Error finding container 12a0d82b88fed864e3bf32b7b41a4af3a37b1c828d858e0de26b002af392910a: Status 404 returned error can't find the container with id 12a0d82b88fed864e3bf32b7b41a4af3a37b1c828d858e0de26b002af392910a Oct 31 18:45:58.588223 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:58.588163 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" event={"ID":"417b4306-1468-44ae-b183-66c9d9a8c330","Type":"ContainerStarted","Data":"12a0d82b88fed864e3bf32b7b41a4af3a37b1c828d858e0de26b002af392910a"} Oct 31 18:45:59.596332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:59.596256 5688 generic.go:356] "Generic (PLEG): container finished" podID="417b4306-1468-44ae-b183-66c9d9a8c330" containerID="061396d823c191a983d25ac4163298a7ee82ec2c82f4fdd70a069a16028127e8" exitCode=0 Oct 31 18:45:59.596665 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:45:59.596339 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" event={"ID":"417b4306-1468-44ae-b183-66c9d9a8c330","Type":"ContainerDied","Data":"061396d823c191a983d25ac4163298a7ee82ec2c82f4fdd70a069a16028127e8"} Oct 31 18:46:00.768261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.768232 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:46:00.916110 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.916026 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn\" (UniqueName: \"kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn\") pod \"417b4306-1468-44ae-b183-66c9d9a8c330\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " Oct 31 18:46:00.916110 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.916101 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca\") pod \"417b4306-1468-44ae-b183-66c9d9a8c330\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " Oct 31 18:46:00.916376 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.916144 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca\") pod \"417b4306-1468-44ae-b183-66c9d9a8c330\" (UID: \"417b4306-1468-44ae-b183-66c9d9a8c330\") " Oct 31 18:46:00.916596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.916565 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca" pod "417b4306-1468-44ae-b183-66c9d9a8c330" (UID: "417b4306-1468-44ae-b183-66c9d9a8c330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:46:00.916764 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.916744 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:46:00.918443 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.918414 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca" pod "417b4306-1468-44ae-b183-66c9d9a8c330" (UID: "417b4306-1468-44ae-b183-66c9d9a8c330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:46:00.918524 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:00.918444 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn" pod "417b4306-1468-44ae-b183-66c9d9a8c330" (UID: "417b4306-1468-44ae-b183-66c9d9a8c330"). InnerVolumeSpecName "kube-api-access-8bknn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:46:01.018306 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:01.018284 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b4306-1468-44ae-b183-66c9d9a8c330-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:46:01.018306 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:01.018306 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bknn\" (UniqueName: \"kubernetes.io/projected/417b4306-1468-44ae-b183-66c9d9a8c330-kube-api-access-8bknn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:46:01.610859 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:01.610833 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" Oct 31 18:46:01.611017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:01.610836 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl" event={"ID":"417b4306-1468-44ae-b183-66c9d9a8c330","Type":"ContainerDied","Data":"12a0d82b88fed864e3bf32b7b41a4af3a37b1c828d858e0de26b002af392910a"} Oct 31 18:46:01.611017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:01.610957 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a0d82b88fed864e3bf32b7b41a4af3a37b1c828d858e0de26b002af392910a" Oct 31 18:46:06.045059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:06.045027 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72"] Oct 31 18:46:06.094358 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:06.094325 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a1ae12-k4t72"] Oct 31 18:46:07.058746 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:07.058702 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd"] Oct 31 18:46:07.102167 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:07.102136 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ad57bc-42pdd"] Oct 31 18:46:07.569197 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:07.569146 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294ca253-4a8a-4f07-8081-5dc30166e590" path="/var/lib/kubelet/pods/294ca253-4a8a-4f07-8081-5dc30166e590/volumes" Oct 31 18:46:07.569735 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:07.569718 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb956115-e3d0-4b10-b0d9-b116b12f809e" path="/var/lib/kubelet/pods/bb956115-e3d0-4b10-b0d9-b116b12f809e/volumes" Oct 31 18:46:15.744178 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:15.744148 5688 generic.go:356] "Generic (PLEG): container finished" podID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerID="6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e" exitCode=0 Oct 31 18:46:15.744599 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:15.744225 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" event={"ID":"f9298d17-2ffe-45c3-b494-4cbfa50e98e9","Type":"ContainerDied","Data":"6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e"} Oct 31 18:46:15.744599 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:15.744562 5688 scope.go:122] "RemoveContainer" containerID="6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e" Oct 31 18:46:15.745893 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:15.745861 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:46:15.746001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:15.745942 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:46:15.746001 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:15.745988 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:46:16.764019 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:16.763979 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" event={"ID":"f9298d17-2ffe-45c3-b494-4cbfa50e98e9","Type":"ContainerStarted","Data":"fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62"} Oct 31 18:46:17.273984 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:17.273538 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:46:17.273984 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:17.273654 5688 patch_prober.go:29] interesting pod/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 18:46:17.273984 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:17.273727 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 18:46:18.273395 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:18.273365 5688 patch_prober.go:29] interesting pod/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 18:46:18.273774 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:18.273416 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 18:46:19.273824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.273788 5688 patch_prober.go:29] interesting pod/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 18:46:19.274311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.273843 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 18:46:19.772493 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.772466 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:46:19.777556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.777529 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:46:19.913411 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.905167 5688 scope.go:122] "RemoveContainer" containerID="6c8195197c56eac0b177c82b210acbff68001b7942c1d319f7f18c58b1f17d72" Oct 31 18:46:19.959523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.959494 5688 scope.go:122] "RemoveContainer" containerID="fde0af346780f83afc4b992230c345f2936d1d852d7c0de4eb9a8668286274d1" Oct 31 18:46:19.968899 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:19.968881 5688 scope.go:122] "RemoveContainer" containerID="ce0b7bc044cd4923d063f2ec33736d6a02c50d82cec8927ace56f3b5eda9ea40" Oct 31 18:46:20.274023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:20.273995 5688 patch_prober.go:29] interesting pod/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 18:46:20.274416 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:20.274050 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 18:46:21.273777 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:21.273746 5688 patch_prober.go:29] interesting pod/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 18:46:21.273949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:21.273807 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 18:46:22.279090 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:22.279058 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:46:22.279646 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:22.279460 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:46:22.281224 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:22.280243 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:46:24.904539 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:46:24.904505 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:46:25.044003 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:25.043971 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7"] Oct 31 18:46:25.093461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:25.093430 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceadf7c-vlvn7"] Oct 31 18:46:25.566864 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:25.566832 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c173ab50-b591-422f-9025-bd1a76a90af5" path="/var/lib/kubelet/pods/c173ab50-b591-422f-9025-bd1a76a90af5/volumes" Oct 31 18:46:29.058805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:29.058769 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp"] Oct 31 18:46:29.112207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:29.107325 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a632e73-rp9zp"] Oct 31 18:46:29.276466 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:29.276434 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm"] Oct 31 18:46:29.620210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:29.604396 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f97e7ab-dbac-417b-89f3-5b5f598286cd" path="/var/lib/kubelet/pods/1f97e7ab-dbac-417b-89f3-5b5f598286cd/volumes" Oct 31 18:46:34.277544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.277491 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" podUID="1226be53-901c-499a-9e15-3dd6d0c29a6f" containerName="payload-pre-processing" containerID="cri-o://5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d" gracePeriod=25 Oct 31 18:46:34.556214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.556176 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:46:34.649915 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.649888 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume\") pod \"1226be53-901c-499a-9e15-3dd6d0c29a6f\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " Oct 31 18:46:34.650057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.649953 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln\" (UniqueName: \"kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln\") pod \"1226be53-901c-499a-9e15-3dd6d0c29a6f\" (UID: \"1226be53-901c-499a-9e15-3dd6d0c29a6f\") " Oct 31 18:46:34.650286 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.650263 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume" pod "1226be53-901c-499a-9e15-3dd6d0c29a6f" (UID: "1226be53-901c-499a-9e15-3dd6d0c29a6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:46:34.650568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.650554 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1226be53-901c-499a-9e15-3dd6d0c29a6f-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:46:34.652056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.652035 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln" pod "1226be53-901c-499a-9e15-3dd6d0c29a6f" (UID: "1226be53-901c-499a-9e15-3dd6d0c29a6f"). InnerVolumeSpecName "kube-api-access-g9jln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:46:34.751509 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.751480 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jln\" (UniqueName: \"kubernetes.io/projected/1226be53-901c-499a-9e15-3dd6d0c29a6f-kube-api-access-g9jln\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:46:34.989307 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.989274 5688 generic.go:356] "Generic (PLEG): container finished" podID="1226be53-901c-499a-9e15-3dd6d0c29a6f" containerID="5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d" exitCode=0 Oct 31 18:46:34.989471 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.989333 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" Oct 31 18:46:34.989471 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.989362 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" event={"ID":"1226be53-901c-499a-9e15-3dd6d0c29a6f","Type":"ContainerDied","Data":"5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d"} Oct 31 18:46:34.989471 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.989444 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm" event={"ID":"1226be53-901c-499a-9e15-3dd6d0c29a6f","Type":"ContainerDied","Data":"3bca1d24300cbef58b72b2825e9bf0323b2cc27350a99ead1f0920a182fe3fc0"} Oct 31 18:46:34.989602 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:34.989471 5688 scope.go:122] "RemoveContainer" containerID="5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d" Oct 31 18:46:35.002537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:35.002440 5688 scope.go:122] "RemoveContainer" containerID="5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d" Oct 31 18:46:35.002942 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:46:35.002922 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d\": container with ID starting with 5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d not found: ID does not exist" containerID="5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d" Oct 31 18:46:35.003004 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:35.002952 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d"} err="failed to get container status \"5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d\": rpc error: code = NotFound desc = could not find container \"5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d\": container with ID starting with 5cddd7438cad6d5d86d4c4b4f769a0483e6e968aad5f83abda3462625afc040d not found: ID does not exist" Oct 31 18:46:35.150737 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:35.150712 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm"] Oct 31 18:46:35.153466 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:35.153447 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-373109-7fcdbc45b7-trnvm"] Oct 31 18:46:35.566269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:35.566240 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1226be53-901c-499a-9e15-3dd6d0c29a6f" path="/var/lib/kubelet/pods/1226be53-901c-499a-9e15-3dd6d0c29a6f/volumes" Oct 31 18:46:38.046549 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:38.046520 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl"] Oct 31 18:46:38.095676 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:38.095649 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00598967-rxfwl"] Oct 31 18:46:39.567723 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:46:39.567686 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457f98ce-f1dd-4e01-a7a8-1148df58d77a" path="/var/lib/kubelet/pods/457f98ce-f1dd-4e01-a7a8-1148df58d77a/volumes" Oct 31 18:46:54.923414 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:46:54.923379 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:47:10.294198 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:10.294091 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp"] Oct 31 18:47:10.294785 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:10.294387 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" containerID="cri-o://5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38" gracePeriod=30 Oct 31 18:47:13.047133 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:13.047099 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr"] Oct 31 18:47:13.096513 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:13.096479 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786aef1f-6cwtr"] Oct 31 18:47:13.580814 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:13.580763 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b9ea6f-c068-4fa5-86bc-165e15c85af7" path="/var/lib/kubelet/pods/26b9ea6f-c068-4fa5-86bc-165e15c85af7/volumes" Oct 31 18:47:15.559855 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.559796 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:47:15.579043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.579019 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r"] Oct 31 18:47:15.579223 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.579201 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" podUID="183949f0-d9c6-4980-9582-01b09818711d" containerName="maas-api" containerID="cri-o://343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea" gracePeriod=30 Oct 31 18:47:15.619818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.619788 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.619929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.619844 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.619929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.619881 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.619929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.619904 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.620093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.619964 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.620093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.619986 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.620093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620018 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.620093 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620079 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.620311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620176 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:15.620311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620243 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data\") pod \"89185377-5b72-49b9-a9f9-97911339f08f\" (UID: \"89185377-5b72-49b9-a9f9-97911339f08f\") " Oct 31 18:47:15.620442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620393 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:15.620502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620444 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:15.620502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620448 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:15.620669 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620647 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:15.620869 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620848 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.620929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620875 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89185377-5b72-49b9-a9f9-97911339f08f-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.620929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620892 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.620929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.620905 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.623120 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.623098 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:15.623120 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.623112 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:47:15.623261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.623156 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "kube-api-access-rwrz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:15.645722 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.645682 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy" pod "89185377-5b72-49b9-a9f9-97911339f08f" (UID: "89185377-5b72-49b9-a9f9-97911339f08f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:15.722315 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.722271 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89185377-5b72-49b9-a9f9-97911339f08f-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.722315 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.722307 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.722529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.722322 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/89185377-5b72-49b9-a9f9-97911339f08f-kube-api-access-rwrz5\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.722529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.722339 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.722529 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.722352 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89185377-5b72-49b9-a9f9-97911339f08f-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:15.848261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.843131 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:47:15.934000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.931449 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs\") pod \"183949f0-d9c6-4980-9582-01b09818711d\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " Oct 31 18:47:15.934000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.931534 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\") pod \"183949f0-d9c6-4980-9582-01b09818711d\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " Oct 31 18:47:15.934000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.931721 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca\") pod \"183949f0-d9c6-4980-9582-01b09818711d\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " Oct 31 18:47:15.934000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.931766 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98\" (UniqueName: \"kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98\") pod \"183949f0-d9c6-4980-9582-01b09818711d\" (UID: \"183949f0-d9c6-4980-9582-01b09818711d\") " Oct 31 18:47:15.934000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.933726 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca" pod "183949f0-d9c6-4980-9582-01b09818711d" (UID: "183949f0-d9c6-4980-9582-01b09818711d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:15.937936 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.937898 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98" pod "183949f0-d9c6-4980-9582-01b09818711d" (UID: "183949f0-d9c6-4980-9582-01b09818711d"). InnerVolumeSpecName "kube-api-access-l4z98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:15.938075 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:15.938022 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs" pod "183949f0-d9c6-4980-9582-01b09818711d" (UID: "183949f0-d9c6-4980-9582-01b09818711d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:47:16.033235 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.026849 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls" pod "183949f0-d9c6-4980-9582-01b09818711d" (UID: "183949f0-d9c6-4980-9582-01b09818711d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:47:16.033418 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.033298 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:16.033418 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.033316 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/183949f0-d9c6-4980-9582-01b09818711d-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:16.033418 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.033325 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/183949f0-d9c6-4980-9582-01b09818711d-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:16.033418 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.033334 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4z98\" (UniqueName: \"kubernetes.io/projected/183949f0-d9c6-4980-9582-01b09818711d-kube-api-access-l4z98\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:16.317157 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.317125 5688 generic.go:356] "Generic (PLEG): container finished" podID="89185377-5b72-49b9-a9f9-97911339f08f" containerID="5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38" exitCode=0 Oct 31 18:47:16.317369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.317225 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" Oct 31 18:47:16.317369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.317232 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" event={"ID":"89185377-5b72-49b9-a9f9-97911339f08f","Type":"ContainerDied","Data":"5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38"} Oct 31 18:47:16.317369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.317311 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp" event={"ID":"89185377-5b72-49b9-a9f9-97911339f08f","Type":"ContainerDied","Data":"fea1b39b1a479ea6c3dbfd527f5fd483d4bdad2cb88253d1a3ce2260b32e2659"} Oct 31 18:47:16.317369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.317342 5688 scope.go:122] "RemoveContainer" containerID="5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38" Oct 31 18:47:16.318933 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.318909 5688 generic.go:356] "Generic (PLEG): container finished" podID="183949f0-d9c6-4980-9582-01b09818711d" containerID="343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea" exitCode=0 Oct 31 18:47:16.319059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.318943 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" event={"ID":"183949f0-d9c6-4980-9582-01b09818711d","Type":"ContainerDied","Data":"343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea"} Oct 31 18:47:16.319059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.318978 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" event={"ID":"183949f0-d9c6-4980-9582-01b09818711d","Type":"ContainerDied","Data":"1a6290b0370602272147103daaa71f3c240be1ff0ebef6cf5e7ba78fd66d56b6"} Oct 31 18:47:16.319059 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.318987 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r" Oct 31 18:47:16.335629 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.335613 5688 scope.go:122] "RemoveContainer" containerID="5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38" Oct 31 18:47:16.335925 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:16.335906 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38\": container with ID starting with 5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38 not found: ID does not exist" containerID="5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38" Oct 31 18:47:16.335987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.335932 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38"} err="failed to get container status \"5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38\": rpc error: code = NotFound desc = could not find container \"5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38\": container with ID starting with 5b854d08a25ca1aefb71b62bb0a8d4f53f62bb2e68b1246f60547937cf8cce38 not found: ID does not exist" Oct 31 18:47:16.335987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.335950 5688 scope.go:122] "RemoveContainer" containerID="343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea" Oct 31 18:47:16.344616 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.344599 5688 scope.go:122] "RemoveContainer" containerID="343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea" Oct 31 18:47:16.344901 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:16.344880 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea\": container with ID starting with 343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea not found: ID does not exist" containerID="343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea" Oct 31 18:47:16.344949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.344908 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea"} err="failed to get container status \"343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea\": rpc error: code = NotFound desc = could not find container \"343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea\": container with ID starting with 343e72bbe47b45862ce974ac72653cef52b24d1cbe3c804b87bba5f287a081ea not found: ID does not exist" Oct 31 18:47:16.529268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.517737 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r"] Oct 31 18:47:16.529268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.520340 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6c8da-7ccbbd748c-q9h7r"] Oct 31 18:47:16.635663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.635573 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp"] Oct 31 18:47:16.638374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.638347 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-ppljp"] Oct 31 18:47:16.942926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.942849 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk"] Oct 31 18:47:16.943103 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:16.943076 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" podUID="0f49ffa2-202c-4dad-8608-8223ea8ff4e1" containerName="maas-api" containerID="cri-o://c0287e0e43d6f90bdc699fd5c702114617ab02b4a6287cf2f44514eda2e02175" gracePeriod=30 Oct 31 18:47:17.339081 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.339045 5688 generic.go:356] "Generic (PLEG): container finished" podID="0f49ffa2-202c-4dad-8608-8223ea8ff4e1" containerID="c0287e0e43d6f90bdc699fd5c702114617ab02b4a6287cf2f44514eda2e02175" exitCode=0 Oct 31 18:47:17.339250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.339116 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" event={"ID":"0f49ffa2-202c-4dad-8608-8223ea8ff4e1","Type":"ContainerDied","Data":"c0287e0e43d6f90bdc699fd5c702114617ab02b4a6287cf2f44514eda2e02175"} Oct 31 18:47:17.339250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.339202 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" event={"ID":"0f49ffa2-202c-4dad-8608-8223ea8ff4e1","Type":"ContainerDied","Data":"7042bd83dd24fbb223d69c22f10d0ddf7b6536c581191b3cc736bf12bde3c55b"} Oct 31 18:47:17.339250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.339232 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7042bd83dd24fbb223d69c22f10d0ddf7b6536c581191b3cc736bf12bde3c55b" Oct 31 18:47:17.351196 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.351162 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:47:17.371376 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.371351 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59\" (UniqueName: \"kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59\") pod \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " Oct 31 18:47:17.371552 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.371534 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca\") pod \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " Oct 31 18:47:17.371619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.371568 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\") pod \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " Oct 31 18:47:17.371673 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.371632 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs\") pod \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\" (UID: \"0f49ffa2-202c-4dad-8608-8223ea8ff4e1\") " Oct 31 18:47:17.372034 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.371840 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca" pod "0f49ffa2-202c-4dad-8608-8223ea8ff4e1" (UID: "0f49ffa2-202c-4dad-8608-8223ea8ff4e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:17.372210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.372163 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:17.374028 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.374006 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59" pod "0f49ffa2-202c-4dad-8608-8223ea8ff4e1" (UID: "0f49ffa2-202c-4dad-8608-8223ea8ff4e1"). InnerVolumeSpecName "kube-api-access-4sc59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:17.374490 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.374470 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs" pod "0f49ffa2-202c-4dad-8608-8223ea8ff4e1" (UID: "0f49ffa2-202c-4dad-8608-8223ea8ff4e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:47:17.374490 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.374484 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls" pod "0f49ffa2-202c-4dad-8608-8223ea8ff4e1" (UID: "0f49ffa2-202c-4dad-8608-8223ea8ff4e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:47:17.475002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.474961 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sc59\" (UniqueName: \"kubernetes.io/projected/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-kube-api-access-4sc59\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:17.475002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.475002 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:17.475369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.475017 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f49ffa2-202c-4dad-8608-8223ea8ff4e1-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:17.580341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.580242 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183949f0-d9c6-4980-9582-01b09818711d" path="/var/lib/kubelet/pods/183949f0-d9c6-4980-9582-01b09818711d/volumes" Oct 31 18:47:17.580830 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:17.580810 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89185377-5b72-49b9-a9f9-97911339f08f" path="/var/lib/kubelet/pods/89185377-5b72-49b9-a9f9-97911339f08f/volumes" Oct 31 18:47:18.348719 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:18.348689 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk" Oct 31 18:47:18.511645 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:18.511616 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk"] Oct 31 18:47:18.561208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:18.561151 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-373109-7fb49c6ccd-vmqmk"] Oct 31 18:47:19.642129 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.642094 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f49ffa2-202c-4dad-8608-8223ea8ff4e1" path="/var/lib/kubelet/pods/0f49ffa2-202c-4dad-8608-8223ea8ff4e1/volumes" Oct 31 18:47:19.942603 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.942512 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8"] Oct 31 18:47:19.943239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943214 5688 cpu_manager.go:414] "containerMap: removing container" podUID="183949f0-d9c6-4980-9582-01b09818711d" containerName="maas-api" Oct 31 18:47:19.943239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943238 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183949f0-d9c6-4980-9582-01b09818711d" containerName="maas-api" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943258 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0f49ffa2-202c-4dad-8608-8223ea8ff4e1" containerName="maas-api" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943264 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f49ffa2-202c-4dad-8608-8223ea8ff4e1" containerName="maas-api" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943275 5688 cpu_manager.go:414] "containerMap: removing container" podUID="417b4306-1468-44ae-b183-66c9d9a8c330" containerName="revoke-keys" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943283 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417b4306-1468-44ae-b183-66c9d9a8c330" containerName="revoke-keys" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943302 5688 cpu_manager.go:414] "containerMap: removing container" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943308 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943323 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1226be53-901c-499a-9e15-3dd6d0c29a6f" containerName="payload-pre-processing" Oct 31 18:47:19.943460 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943330 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1226be53-901c-499a-9e15-3dd6d0c29a6f" containerName="payload-pre-processing" Oct 31 18:47:19.943817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943514 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f49ffa2-202c-4dad-8608-8223ea8ff4e1" containerName="maas-api" Oct 31 18:47:19.943817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943530 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="89185377-5b72-49b9-a9f9-97911339f08f" containerName="istio-proxy" Oct 31 18:47:19.943817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943540 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1226be53-901c-499a-9e15-3dd6d0c29a6f" containerName="payload-pre-processing" Oct 31 18:47:19.943817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943558 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="183949f0-d9c6-4980-9582-01b09818711d" containerName="maas-api" Oct 31 18:47:19.943817 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.943567 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="417b4306-1468-44ae-b183-66c9d9a8c330" containerName="revoke-keys" Oct 31 18:47:19.973208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:19.970707 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.023208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.020260 5688 scope.go:122] "RemoveContainer" containerID="8de1c788fe6ed8489642fa39ad8ed033a4d8b495c7eaa2577c02bf4317ac90a7" Oct 31 18:47:20.030675 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.030648 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8"] Oct 31 18:47:20.060097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.060060 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-66032c-687ccdf449-94wt8\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.060556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.060529 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfrx\" (UniqueName: \"kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx\") pod \"payload-processing-e2e-api-b-66032c-687ccdf449-94wt8\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.061050 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.061024 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-66032c-dockercfg-tr5q5\"" Oct 31 18:47:20.061228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.061213 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-66032c\"" Oct 31 18:47:20.134123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.134099 5688 scope.go:122] "RemoveContainer" containerID="57c2a86bbb063adcf08e22edce800c171015f61cbf30212140a75fc7812f5d58" Oct 31 18:47:20.154727 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.154695 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97"] Oct 31 18:47:20.161099 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.161075 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.161900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.161871 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfrx\" (UniqueName: \"kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx\") pod \"payload-processing-e2e-api-b-66032c-687ccdf449-94wt8\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.162530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.162505 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-66032c-687ccdf449-94wt8\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.163440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.163421 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-66032c-687ccdf449-94wt8\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.166506 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.165444 5688 scope.go:122] "RemoveContainer" containerID="099131dc6b4bd3d36f14004bdd7724f8b927c0f0d77b8faa18948e880dcacb4a" Oct 31 18:47:20.180841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.180826 5688 scope.go:122] "RemoveContainer" containerID="40c1a43a4f8cfbd93c61aea93c8c522151f84058dd81cc5d6f5972d86c4bb6eb" Oct 31 18:47:20.189535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.189517 5688 scope.go:122] "RemoveContainer" containerID="53f436ad491608a6390b1761c0773dc7fcf3ceb7aec6af46c9dc5ae426a8abf0" Oct 31 18:47:20.213737 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.213714 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97"] Oct 31 18:47:20.218540 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.218515 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfrx\" (UniqueName: \"kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx\") pod \"payload-processing-e2e-api-b-66032c-687ccdf449-94wt8\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.264681 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.264657 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.264876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.264853 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55cg5\" (UniqueName: \"kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5\") pod \"payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.320340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.320302 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:20.365806 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.365768 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55cg5\" (UniqueName: \"kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5\") pod \"payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.365995 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.365979 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.366666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.366647 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.426782 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.422541 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27"] Oct 31 18:47:20.427498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.427473 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55cg5\" (UniqueName: \"kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5\") pod \"payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.428064 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.428050 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.480258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.480234 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:20.480762 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.480738 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 18:47:20.480869 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.480803 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 18:47:20.482771 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.482742 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27"] Oct 31 18:47:20.569246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.569216 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.569412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.569389 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42pc\" (UniqueName: \"kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.569538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.569472 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.670812 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.670780 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42pc\" (UniqueName: \"kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.671237 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.670874 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.671237 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.670923 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.671635 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.671618 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.673362 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.673345 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.705339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.705312 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8"] Oct 31 18:47:20.706600 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:47:20.706578 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069e7f65_9c46_4dbf_9465_7bd4bcedf63e.slice/crio-cab5352170c11cb7040e138ecde9ddd68272335a9ae3c39567973f75faaa75fe WatchSource:0}: Error finding container cab5352170c11cb7040e138ecde9ddd68272335a9ae3c39567973f75faaa75fe: Status 404 returned error can't find the container with id cab5352170c11cb7040e138ecde9ddd68272335a9ae3c39567973f75faaa75fe Oct 31 18:47:20.737414 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.737395 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42pc\" (UniqueName: \"kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc\") pod \"maas-api-revoke-sub-f5ddd3ef-v8n27\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.759484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.759464 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:20.872551 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:20.872517 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97"] Oct 31 18:47:21.149422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.149384 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27"] Oct 31 18:47:21.151674 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:47:21.151642 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248a9706_8640_4ad6_84fd_183182afaf67.slice/crio-bcacc0d4eb55e0c06fb27c9a6570342a8dde0fe21d08d5c72b826e5d019caad8 WatchSource:0}: Error finding container bcacc0d4eb55e0c06fb27c9a6570342a8dde0fe21d08d5c72b826e5d019caad8: Status 404 returned error can't find the container with id bcacc0d4eb55e0c06fb27c9a6570342a8dde0fe21d08d5c72b826e5d019caad8 Oct 31 18:47:21.368938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.368906 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" event={"ID":"d87787af-4e3c-4fd1-a3a4-918505116192","Type":"ContainerStarted","Data":"97c3977835dd93b6ee9676052d6ab8971b1138368e5261cd26c59300871e1e05"} Oct 31 18:47:21.369081 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.368976 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:21.369081 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.369005 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" event={"ID":"d87787af-4e3c-4fd1-a3a4-918505116192","Type":"ContainerStarted","Data":"721e90b6f7c67e998d08d95fe96d0b0888066a2a5418239480ce207aae7c0e5b"} Oct 31 18:47:21.370229 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.370174 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" event={"ID":"248a9706-8640-4ad6-84fd-183182afaf67","Type":"ContainerStarted","Data":"bcacc0d4eb55e0c06fb27c9a6570342a8dde0fe21d08d5c72b826e5d019caad8"} Oct 31 18:47:21.371742 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.371720 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" event={"ID":"069e7f65-9c46-4dbf-9465-7bd4bcedf63e","Type":"ContainerStarted","Data":"61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb"} Oct 31 18:47:21.371875 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.371757 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" event={"ID":"069e7f65-9c46-4dbf-9465-7bd4bcedf63e","Type":"ContainerStarted","Data":"cab5352170c11cb7040e138ecde9ddd68272335a9ae3c39567973f75faaa75fe"} Oct 31 18:47:21.371875 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.371844 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:21.483087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.483047 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" podStartSLOduration=1.483032742 podStartE2EDuration="1.483032742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:21.482118896 +0000 UTC m=+1562.084260040" watchObservedRunningTime="2026-10-31 18:47:21.483032742 +0000 UTC m=+1562.085173886" Oct 31 18:47:21.613741 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:21.609046 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" podStartSLOduration=2.60902355 podStartE2EDuration="2.60902355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:21.608088612 +0000 UTC m=+1562.210229760" watchObservedRunningTime="2026-10-31 18:47:21.60902355 +0000 UTC m=+1562.211164722" Oct 31 18:47:22.039952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:22.039915 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f"] Oct 31 18:47:22.042674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:22.042652 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccf13ff-9wb7f"] Oct 31 18:47:22.379461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:22.379379 5688 generic.go:356] "Generic (PLEG): container finished" podID="248a9706-8640-4ad6-84fd-183182afaf67" containerID="153a85dacefb75ec9032ea068cfbf8706859484e3cebe8b329db54c6d0a66783" exitCode=0 Oct 31 18:47:22.379461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:22.379442 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" event={"ID":"248a9706-8640-4ad6-84fd-183182afaf67","Type":"ContainerDied","Data":"153a85dacefb75ec9032ea068cfbf8706859484e3cebe8b329db54c6d0a66783"} Oct 31 18:47:23.586226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.580000 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d83f600-e654-4da1-bb65-c4ff81f0c38d" path="/var/lib/kubelet/pods/0d83f600-e654-4da1-bb65-c4ff81f0c38d/volumes" Oct 31 18:47:23.640409 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.640381 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:23.712618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.712542 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc\" (UniqueName: \"kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc\") pod \"248a9706-8640-4ad6-84fd-183182afaf67\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " Oct 31 18:47:23.712772 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.712752 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca\") pod \"248a9706-8640-4ad6-84fd-183182afaf67\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " Oct 31 18:47:23.712851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.712835 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca\") pod \"248a9706-8640-4ad6-84fd-183182afaf67\" (UID: \"248a9706-8640-4ad6-84fd-183182afaf67\") " Oct 31 18:47:23.713086 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.713065 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca" pod "248a9706-8640-4ad6-84fd-183182afaf67" (UID: "248a9706-8640-4ad6-84fd-183182afaf67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:23.713395 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.713374 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:23.714727 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.714704 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc" pod "248a9706-8640-4ad6-84fd-183182afaf67" (UID: "248a9706-8640-4ad6-84fd-183182afaf67"). InnerVolumeSpecName "kube-api-access-k42pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:23.714881 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.714862 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca" pod "248a9706-8640-4ad6-84fd-183182afaf67" (UID: "248a9706-8640-4ad6-84fd-183182afaf67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:47:23.814540 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.814510 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42pc\" (UniqueName: \"kubernetes.io/projected/248a9706-8640-4ad6-84fd-183182afaf67-kube-api-access-k42pc\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:23.814540 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:23.814536 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248a9706-8640-4ad6-84fd-183182afaf67-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:24.434485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:24.434388 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" Oct 31 18:47:24.434943 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:24.434913 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27" event={"ID":"248a9706-8640-4ad6-84fd-183182afaf67","Type":"ContainerDied","Data":"bcacc0d4eb55e0c06fb27c9a6570342a8dde0fe21d08d5c72b826e5d019caad8"} Oct 31 18:47:24.435254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:24.435233 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcacc0d4eb55e0c06fb27c9a6570342a8dde0fe21d08d5c72b826e5d019caad8" Oct 31 18:47:24.904524 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:24.904489 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:47:25.846924 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:25.846887 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq"] Oct 31 18:47:25.847247 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:25.847216 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" containerID="cri-o://bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a" gracePeriod=30 Oct 31 18:47:27.046058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:27.046023 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf"] Oct 31 18:47:27.095444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:27.095413 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb6bb94-kxnqf"] Oct 31 18:47:27.568008 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:27.567976 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3c986b-530c-4205-8180-dd32b661530d" path="/var/lib/kubelet/pods/cc3c986b-530c-4205-8180-dd32b661530d/volumes" Oct 31 18:47:27.989826 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:27.989794 5688 patch_prober.go:29] interesting pod/e2e-api-b-373109-openshift-default-844b974c49-88jqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:47:27.989982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:27.989856 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:47:28.732647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:28.732620 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm"] Oct 31 18:47:28.733075 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:28.732866 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" containerID="cri-o://b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb" gracePeriod=30 Oct 31 18:47:31.119795 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.119739 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:47:31.152923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.152900 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.152934 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.152953 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.152975 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153013 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153082 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153234 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153266 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153298 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:31.153456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153357 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:31.153456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153368 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo\") pod \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\" (UID: \"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba\") " Oct 31 18:47:31.153456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153408 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:31.153663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153508 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:31.153663 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153540 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:31.153889 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153867 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.154023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153890 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.154023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153900 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.154023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153914 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.154023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.153926 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.155516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.155483 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:47:31.155619 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.155573 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "kube-api-access-f7rb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:31.155710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.155691 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:31.155772 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.155753 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token" pod "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" (UID: "d0da9986-1e4d-4af6-8b60-dc9ee793d4ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:31.255568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.255541 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.255568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.255564 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.255726 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.255574 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7rb4\" (UniqueName: \"kubernetes.io/projected/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-kube-api-access-f7rb4\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.255726 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.255584 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:31.519257 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.519222 5688 generic.go:356] "Generic (PLEG): container finished" podID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerID="bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a" exitCode=0 Oct 31 18:47:31.519400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.519306 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" Oct 31 18:47:31.519400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.519340 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" event={"ID":"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba","Type":"ContainerDied","Data":"bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a"} Oct 31 18:47:31.519481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.519427 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq" event={"ID":"d0da9986-1e4d-4af6-8b60-dc9ee793d4ba","Type":"ContainerDied","Data":"1ef89a2b2af7ecb0192f7c413936bbd4c276d1c426b8ea4bd686000cc2923d84"} Oct 31 18:47:31.519481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.519456 5688 scope.go:122] "RemoveContainer" containerID="bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a" Oct 31 18:47:31.530303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.530286 5688 scope.go:122] "RemoveContainer" containerID="5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292" Oct 31 18:47:31.538994 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.538979 5688 scope.go:122] "RemoveContainer" containerID="bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a" Oct 31 18:47:31.539245 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:31.539222 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a\": container with ID starting with bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a not found: ID does not exist" containerID="bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a" Oct 31 18:47:31.539321 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.539252 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a"} err="failed to get container status \"bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a\": rpc error: code = NotFound desc = could not find container \"bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a\": container with ID starting with bb686ca1b08cb1c59fee6ab0eb3cc641a79a2e271ac8bf3c7a6f029f021a122a not found: ID does not exist" Oct 31 18:47:31.539321 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.539270 5688 scope.go:122] "RemoveContainer" containerID="5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292" Oct 31 18:47:31.539488 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:31.539474 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292\": container with ID starting with 5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292 not found: ID does not exist" containerID="5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292" Oct 31 18:47:31.539524 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.539491 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292"} err="failed to get container status \"5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292\": rpc error: code = NotFound desc = could not find container \"5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292\": container with ID starting with 5dd8935474fc948403f5c2a228834965313ad18d6da6aca8b93c579a14310292 not found: ID does not exist" Oct 31 18:47:31.682568 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.682536 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq"] Oct 31 18:47:31.686755 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:31.686732 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-373109-openshift-default-844b974c49-88jqq"] Oct 31 18:47:32.382885 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:32.382858 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:47:32.383348 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:32.382938 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:47:32.563737 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:32.563710 5688 patch_prober.go:29] interesting pod/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:47:32.563881 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:32.563753 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:47:33.566372 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:33.566341 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" path="/var/lib/kubelet/pods/d0da9986-1e4d-4af6-8b60-dc9ee793d4ba/volumes" Oct 31 18:47:34.013835 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.013809 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:47:34.110923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.110856 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.110923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.110914 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.110966 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.110982 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111013 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111059 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111089 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111133 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:34.111429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111150 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111215 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token\") pod \"279d78ee-f449-4b03-8ef6-2b90eb588733\" (UID: \"279d78ee-f449-4b03-8ef6-2b90eb588733\") " Oct 31 18:47:34.111429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111307 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:34.111650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111441 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:34.111650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111456 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:34.111650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111485 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:34.111802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111676 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.111802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111690 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.111802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111699 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279d78ee-f449-4b03-8ef6-2b90eb588733-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.111802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111713 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.111802 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.111723 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.113547 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.113521 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "kube-api-access-wt5vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:34.113666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.113555 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:47:34.113666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.113561 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:34.113776 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.113724 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token" pod "279d78ee-f449-4b03-8ef6-2b90eb588733" (UID: "279d78ee-f449-4b03-8ef6-2b90eb588733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:34.213477 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.213441 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.213477 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.213476 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.213671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.213490 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.213671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.213506 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt5vj\" (UniqueName: \"kubernetes.io/projected/279d78ee-f449-4b03-8ef6-2b90eb588733-kube-api-access-wt5vj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:34.595255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.595159 5688 generic.go:356] "Generic (PLEG): container finished" podID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerID="b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb" exitCode=0 Oct 31 18:47:34.596546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.595921 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" event={"ID":"279d78ee-f449-4b03-8ef6-2b90eb588733","Type":"ContainerDied","Data":"b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb"} Oct 31 18:47:34.596546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.595985 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" event={"ID":"279d78ee-f449-4b03-8ef6-2b90eb588733","Type":"ContainerDied","Data":"32c8758d06b711a20fbe7d3f528b4708b2921946eb4ab2fb6960fcbbd4951eb5"} Oct 31 18:47:34.596546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.596015 5688 scope.go:122] "RemoveContainer" containerID="b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb" Oct 31 18:47:34.596546 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.596144 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm" Oct 31 18:47:34.625599 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.625575 5688 scope.go:122] "RemoveContainer" containerID="0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca" Oct 31 18:47:34.634859 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.634837 5688 scope.go:122] "RemoveContainer" containerID="b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb" Oct 31 18:47:34.635118 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:34.635100 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb\": container with ID starting with b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb not found: ID does not exist" containerID="b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb" Oct 31 18:47:34.635163 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.635129 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb"} err="failed to get container status \"b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb\": rpc error: code = NotFound desc = could not find container \"b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb\": container with ID starting with b176aae9daf7343e6bf5b913b5896ae2c8173847540ff84b7f574e94984fcefb not found: ID does not exist" Oct 31 18:47:34.635163 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.635149 5688 scope.go:122] "RemoveContainer" containerID="0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca" Oct 31 18:47:34.635413 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:34.635396 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca\": container with ID starting with 0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca not found: ID does not exist" containerID="0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca" Oct 31 18:47:34.635449 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.635420 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca"} err="failed to get container status \"0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca\": rpc error: code = NotFound desc = could not find container \"0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca\": container with ID starting with 0b3c513884fda480b0ea9adf8df2eb7284829b9fa07eee3afabda02ce1cabdca not found: ID does not exist" Oct 31 18:47:34.779335 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.778941 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm"] Oct 31 18:47:34.783201 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:34.783156 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a6c8da-openshift-default-fb7448c55-klxcm"] Oct 31 18:47:35.567215 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:35.567153 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" path="/var/lib/kubelet/pods/279d78ee-f449-4b03-8ef6-2b90eb588733/volumes" Oct 31 18:47:39.110560 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:39.110528 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq"] Oct 31 18:47:39.110983 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:39.110742 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" containerID="cri-o://47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3" gracePeriod=30 Oct 31 18:47:39.160367 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:39.160341 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr"] Oct 31 18:47:39.160584 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:39.160538 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" containerID="cri-o://4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54" gracePeriod=30 Oct 31 18:47:43.980480 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:43.980448 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:47:43.980888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:43.980501 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:47:44.437321 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.437295 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:47:44.463960 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.463934 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:47:44.549138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549048 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549095 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549138 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549125 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549156 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549237 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549282 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549305 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549344 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549376 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549390 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549415 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549451 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549481 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549520 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549573 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549599 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549656 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549793 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs\") pod \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\" (UID: \"3b48bcae-d1cd-4ab6-87c0-fae1080e6227\") " Oct 31 18:47:44.549901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.549847 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data\") pod \"b8568b17-5b99-416e-81f7-ee05f6a471ef\" (UID: \"b8568b17-5b99-416e-81f7-ee05f6a471ef\") " Oct 31 18:47:44.550293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.550121 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.550421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.550395 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.550516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.550459 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.550599 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.550697 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.550880 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551099 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551126 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551141 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551156 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8568b17-5b99-416e-81f7-ee05f6a471ef-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551165 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551171 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551226 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.551374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551245 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.552002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.551530 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.552232 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.552147 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.552922 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.552882 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:44.553341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.553307 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:47:44.553683 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.553657 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "kube-api-access-tdrbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:44.553767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.553721 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:44.553825 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.553807 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "kube-api-access-qxkmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:47:44.553952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.553934 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy" pod "b8568b17-5b99-416e-81f7-ee05f6a471ef" (UID: "b8568b17-5b99-416e-81f7-ee05f6a471ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.554333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.554307 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:47:44.554982 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.554962 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy" pod "3b48bcae-d1cd-4ab6-87c0-fae1080e6227" (UID: "3b48bcae-d1cd-4ab6-87c0-fae1080e6227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:47:44.652293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652261 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652290 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652304 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdrbz\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-kube-api-access-tdrbz\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652317 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652332 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652344 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkmr\" (UniqueName: \"kubernetes.io/projected/b8568b17-5b99-416e-81f7-ee05f6a471ef-kube-api-access-qxkmr\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652357 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652369 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652382 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8568b17-5b99-416e-81f7-ee05f6a471ef-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652394 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.652517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.652406 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b48bcae-d1cd-4ab6-87c0-fae1080e6227-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:47:44.825227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.825115 5688 generic.go:356] "Generic (PLEG): container finished" podID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerID="47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3" exitCode=0 Oct 31 18:47:44.825227 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.825162 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" Oct 31 18:47:44.825419 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.825225 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" event={"ID":"3b48bcae-d1cd-4ab6-87c0-fae1080e6227","Type":"ContainerDied","Data":"47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3"} Oct 31 18:47:44.825419 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.825278 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq" event={"ID":"3b48bcae-d1cd-4ab6-87c0-fae1080e6227","Type":"ContainerDied","Data":"acfca5fa1d69bcf8e483ac5210bee6ed14dfe05b526db8c2b2a739a9bcf90e94"} Oct 31 18:47:44.825419 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.825307 5688 scope.go:122] "RemoveContainer" containerID="47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3" Oct 31 18:47:44.827029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.827007 5688 generic.go:356] "Generic (PLEG): container finished" podID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerID="4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54" exitCode=0 Oct 31 18:47:44.827152 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.827058 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" Oct 31 18:47:44.827152 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.827133 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" event={"ID":"b8568b17-5b99-416e-81f7-ee05f6a471ef","Type":"ContainerDied","Data":"4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54"} Oct 31 18:47:44.827296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.827172 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" event={"ID":"b8568b17-5b99-416e-81f7-ee05f6a471ef","Type":"ContainerDied","Data":"cfea9614d6a4b60b7ebe11193c32067fb26ea8fd47b036afe6c9a6abdc44369e"} Oct 31 18:47:44.878238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.869606 5688 scope.go:122] "RemoveContainer" containerID="47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3" Oct 31 18:47:44.878238 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:44.870964 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3\": container with ID starting with 47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3 not found: ID does not exist" containerID="47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3" Oct 31 18:47:44.878238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.871005 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3"} err="failed to get container status \"47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3\": rpc error: code = NotFound desc = could not find container \"47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3\": container with ID starting with 47f51cb28a2e8c51202c2ede94470038d46e3c47570cd2ceefb0917de4f494b3 not found: ID does not exist" Oct 31 18:47:44.878238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.871023 5688 scope.go:122] "RemoveContainer" containerID="4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54" Oct 31 18:47:44.907256 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.907226 5688 scope.go:122] "RemoveContainer" containerID="4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54" Oct 31 18:47:44.914639 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:44.914592 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54\": container with ID starting with 4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54 not found: ID does not exist" containerID="4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54" Oct 31 18:47:44.914639 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.914642 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54"} err="failed to get container status \"4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54\": rpc error: code = NotFound desc = could not find container \"4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54\": container with ID starting with 4532d3d29b744175b090e9ffb511346c70ff03f1fba44296323999e3c9946a54 not found: ID does not exist" Oct 31 18:47:44.989651 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.989623 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq"] Oct 31 18:47:44.993369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:44.993342 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-n76lq"] Oct 31 18:47:45.101156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:45.101086 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr"] Oct 31 18:47:45.104240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:45.104219 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr"] Oct 31 18:47:45.439237 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:45.439161 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:47:45.439369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:45.439263 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-rl4pr" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:47:45.565742 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:45.565709 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" path="/var/lib/kubelet/pods/3b48bcae-d1cd-4ab6-87c0-fae1080e6227/volumes" Oct 31 18:47:45.566240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:47:45.566221 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" path="/var/lib/kubelet/pods/b8568b17-5b99-416e-81f7-ee05f6a471ef/volumes" Oct 31 18:47:54.904720 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:47:54.904687 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:48:04.191208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191158 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh"] Oct 31 18:48:04.191659 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191644 5688 cpu_manager.go:414] "containerMap: removing container" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" Oct 31 18:48:04.191702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191660 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" Oct 31 18:48:04.191702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191679 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" Oct 31 18:48:04.191702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191684 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191709 5688 cpu_manager.go:414] "containerMap: removing container" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191716 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191722 5688 cpu_manager.go:414] "containerMap: removing container" podUID="248a9706-8640-4ad6-84fd-183182afaf67" containerName="revoke-keys" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191727 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248a9706-8640-4ad6-84fd-183182afaf67" containerName="revoke-keys" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191734 5688 cpu_manager.go:414] "containerMap: removing container" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191738 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191753 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191757 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191764 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" Oct 31 18:48:04.191809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191769 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" Oct 31 18:48:04.192112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191846 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8568b17-5b99-416e-81f7-ee05f6a471ef" containerName="istio-proxy" Oct 31 18:48:04.192112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191854 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" Oct 31 18:48:04.192112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191860 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" Oct 31 18:48:04.192112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191867 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0da9986-1e4d-4af6-8b60-dc9ee793d4ba" containerName="istio-proxy" Oct 31 18:48:04.192112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191875 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b48bcae-d1cd-4ab6-87c0-fae1080e6227" containerName="istio-proxy" Oct 31 18:48:04.192112 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.191881 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="248a9706-8640-4ad6-84fd-183182afaf67" containerName="revoke-keys" Oct 31 18:48:04.196872 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.196854 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.298710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.298685 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-66032c-serving-cert\"" Oct 31 18:48:04.301287 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.301266 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh"] Oct 31 18:48:04.310316 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.310296 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.310380 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.310347 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.310447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.310429 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.310481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.310456 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.411993 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.411960 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.412141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.412022 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.412141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.412060 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.412141 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.412084 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.412297 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:04.412213 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-serving-cert: secret "maas-api-e2e-api-b-66032c-serving-cert" not found Oct 31 18:48:04.412297 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:04.412276 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls podName:9390acf7-e67b-4306-a393-a1a242e77c46 nodeName:}" failed. No retries permitted until 2026-10-31 18:48:04.912257297 +0000 UTC m=+1605.514398421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls") pod "maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" (UID: "9390acf7-e67b-4306-a393-a1a242e77c46") : secret "maas-api-e2e-api-b-66032c-serving-cert" not found Oct 31 18:48:04.412757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.412736 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.414496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.414475 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.483887 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.483290 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.924177 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.924098 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:04.926714 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:04.926688 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\") pod \"maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:05.110506 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:05.110472 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:05.485689 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:05.485663 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh"] Oct 31 18:48:05.487336 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:48:05.487307 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9390acf7_e67b_4306_a393_a1a242e77c46.slice/crio-6c80e990bd0d632e7e18b602fdbfa3df03487db6a7f4774ac8bfb1b9b9479b1a WatchSource:0}: Error finding container 6c80e990bd0d632e7e18b602fdbfa3df03487db6a7f4774ac8bfb1b9b9479b1a: Status 404 returned error can't find the container with id 6c80e990bd0d632e7e18b602fdbfa3df03487db6a7f4774ac8bfb1b9b9479b1a Oct 31 18:48:05.996092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:05.996008 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" event={"ID":"9390acf7-e67b-4306-a393-a1a242e77c46","Type":"ContainerStarted","Data":"6c80e990bd0d632e7e18b602fdbfa3df03487db6a7f4774ac8bfb1b9b9479b1a"} Oct 31 18:48:07.004283 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:07.004246 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" event={"ID":"9390acf7-e67b-4306-a393-a1a242e77c46","Type":"ContainerStarted","Data":"f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d"} Oct 31 18:48:07.004648 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:07.004434 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:07.045204 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:07.045150 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6"] Oct 31 18:48:07.094282 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:07.094257 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbe3bb-24zx6"] Oct 31 18:48:07.153276 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:07.153228 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" podStartSLOduration=2.3456833489999998 podStartE2EDuration="3.153214658s" totalImagesPullingTime="807.531309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:07.152693694 +0000 UTC m=+1607.754834834" watchObservedRunningTime="2026-10-31 18:48:07.153214658 +0000 UTC m=+1607.755355833" Oct 31 18:48:07.566940 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:07.566909 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8500cd6b-c5d1-4a31-b1fa-908ad840cf35" path="/var/lib/kubelet/pods/8500cd6b-c5d1-4a31-b1fa-908ad840cf35/volumes" Oct 31 18:48:13.015505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:13.015430 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:48:16.306294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:16.306266 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7"] Oct 31 18:48:16.306766 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:16.306444 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" containerID="cri-o://3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3" gracePeriod=30 Oct 31 18:48:17.633543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.633516 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf"] Oct 31 18:48:17.634075 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.634061 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="279d78ee-f449-4b03-8ef6-2b90eb588733" containerName="istio-proxy" Oct 31 18:48:17.637573 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.637555 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.689650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.689630 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3417b822-openshift-default-dockercfg-hx6x8\"" Oct 31 18:48:17.743409 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.743376 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf"] Oct 31 18:48:17.774121 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774087 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774274 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774256 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774283 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774363 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774316 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774476 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774458 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774554 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774537 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774625 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774697 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774652 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgn7\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.774743 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.774726 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.875570 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875533 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.875723 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875597 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.875723 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875634 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.875723 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875657 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgn7\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.875909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875791 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.875909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875858 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876015 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.875989 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876068 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876023 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876068 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876040 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876067 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876158 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876316 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876225 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876439 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.876736 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.876717 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.878228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.878209 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.878683 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.878664 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.931618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.931559 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.931818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.931798 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgn7\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:17.950629 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:17.950611 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:18.375958 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:18.375927 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf"] Oct 31 18:48:18.417084 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:48:18.417058 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130d7162_286d_4211_9720_67cb863812e3.slice/crio-bf5092f11484e619f175745905cf10bef853939dde326ebf3725e80addc2eba8 WatchSource:0}: Error finding container bf5092f11484e619f175745905cf10bef853939dde326ebf3725e80addc2eba8: Status 404 returned error can't find the container with id bf5092f11484e619f175745905cf10bef853939dde326ebf3725e80addc2eba8 Oct 31 18:48:18.419484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:18.419448 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:18.419572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:18.419520 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:18.419572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:18.419548 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:19.080445 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:19.080402 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" event={"ID":"130d7162-286d-4211-9720-67cb863812e3","Type":"ContainerStarted","Data":"4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400"} Oct 31 18:48:19.080851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:19.080470 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" event={"ID":"130d7162-286d-4211-9720-67cb863812e3","Type":"ContainerStarted","Data":"bf5092f11484e619f175745905cf10bef853939dde326ebf3725e80addc2eba8"} Oct 31 18:48:19.305044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:19.304991 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podStartSLOduration=2.304977353 podStartE2EDuration="2.304977353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:19.255646427 +0000 UTC m=+1619.857787571" watchObservedRunningTime="2026-10-31 18:48:19.304977353 +0000 UTC m=+1619.907118491" Oct 31 18:48:19.953510 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:19.953479 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:19.958553 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:19.958521 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 18:48:19.958684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:19.958581 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 18:48:20.271339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.271311 5688 scope.go:122] "RemoveContainer" containerID="5609f4ba456c9bf06fa571e4586688f92ac421d32021fe73cd7a77e77d0466fe" Oct 31 18:48:20.283690 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.283667 5688 scope.go:122] "RemoveContainer" containerID="da4cb3798e33e7ccb6a598c795920846d8423caf31c76a4349813aa4d4d777c5" Oct 31 18:48:20.291921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.291906 5688 scope.go:122] "RemoveContainer" containerID="2652e7260ab428bbc9b6673e0953f70b627060954dbf86a58c3f7b0972348436" Oct 31 18:48:20.299925 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.299908 5688 scope.go:122] "RemoveContainer" containerID="35af03604280eab54b17ca0da81967e903ed6c0003819a37e92ce3cb8ad8d251" Oct 31 18:48:20.307678 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.307662 5688 scope.go:122] "RemoveContainer" containerID="b29b3e77225d76fcc8551f1420cb9f1947b9238271e2c751ae2d6cb9095209ee" Oct 31 18:48:20.957212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.954523 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 18:48:20.957212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:20.954578 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 18:48:21.601780 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.601748 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:48:21.720613 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720579 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.720787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720621 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.720787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720683 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.720787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720709 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.720946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720800 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.720946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720837 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.720946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720875 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.721097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720969 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.721097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.720999 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket\") pod \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\" (UID: \"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c\") " Oct 31 18:48:21.721097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721007 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:48:21.721286 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721257 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:21.721418 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721395 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:21.721522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721501 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:21.721957 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721920 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.721957 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721947 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.722057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721965 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.722057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.721980 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.722128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.722063 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:21.723147 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.723127 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:21.723503 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.723473 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "kube-api-access-db6fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:48:21.723640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.723521 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:48:21.724598 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.724561 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token" pod "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" (UID: "4c1c2252-f6c0-40e3-8a99-9b20696e0c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:48:21.823070 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.823032 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.823070 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.823067 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db6fs\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-kube-api-access-db6fs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.823308 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.823082 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.823308 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.823095 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.823308 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.823107 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:21.965842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.965743 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 18:48:21.965842 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:21.965825 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 18:48:22.116967 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.116933 5688 generic.go:356] "Generic (PLEG): container finished" podID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerID="3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3" exitCode=0 Oct 31 18:48:22.116967 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.116963 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" event={"ID":"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c","Type":"ContainerDied","Data":"3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3"} Oct 31 18:48:22.117254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.117006 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" event={"ID":"4c1c2252-f6c0-40e3-8a99-9b20696e0c5c","Type":"ContainerDied","Data":"cad14aa0ae9f8087449d888d85c5efc99e2c431a9073607b6edf34fb1ab4c9c7"} Oct 31 18:48:22.117254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.116947 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7" Oct 31 18:48:22.117254 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.117028 5688 scope.go:122] "RemoveContainer" containerID="3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3" Oct 31 18:48:22.127208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.127161 5688 scope.go:122] "RemoveContainer" containerID="3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3" Oct 31 18:48:22.127501 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:22.127483 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3\": container with ID starting with 3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3 not found: ID does not exist" containerID="3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3" Oct 31 18:48:22.127555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.127510 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3"} err="failed to get container status \"3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3\": rpc error: code = NotFound desc = could not find container \"3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3\": container with ID starting with 3a80592269929ad700572162f93f57cdec8ac95b8269ca00c4047df533cf2bc3 not found: ID does not exist" Oct 31 18:48:22.282212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.282165 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7"] Oct 31 18:48:22.330131 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.330105 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-9vdq7"] Oct 31 18:48:22.952214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.951479 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 18:48:22.952214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:22.951533 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 18:48:23.566155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:23.566120 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" path="/var/lib/kubelet/pods/4c1c2252-f6c0-40e3-8a99-9b20696e0c5c/volumes" Oct 31 18:48:23.702023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:23.701987 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5"] Oct 31 18:48:23.702711 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:23.702662 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" containerID="cri-o://fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62" gracePeriod=30 Oct 31 18:48:23.951649 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:23.951557 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 18:48:23.951907 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:23.951880 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 18:48:24.904871 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:24.904829 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:48:25.174945 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:25.174863 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:25.175235 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:25.175214 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:25.175962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:25.175944 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:48:28.960323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.960298 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:48:28.996543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996524 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.996668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996566 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.996668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996595 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.996668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996626 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.996668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996653 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.996926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996784 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.996926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996782 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:28.996926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996840 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:28.996926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996902 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.997126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996935 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.997126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.996985 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data\") pod \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\" (UID: \"f9298d17-2ffe-45c3-b494-4cbfa50e98e9\") " Oct 31 18:48:28.997255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.997218 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:28.997425 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.997402 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:28.997535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.997448 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:28.997535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.997495 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:28.997535 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.997511 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:28.997653 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.997517 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:48:28.998944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.998922 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:48:28.999088 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.999055 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:48:28.999482 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.999449 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:48:28.999587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:28.999495 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76" pod "f9298d17-2ffe-45c3-b494-4cbfa50e98e9" (UID: "f9298d17-2ffe-45c3-b494-4cbfa50e98e9"). InnerVolumeSpecName "kube-api-access-7mh76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:48:29.099384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.099351 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:29.099384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.099386 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:29.099556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.099401 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:29.099556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.099414 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:29.099556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.099428 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:29.099556 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.099440 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mh76\" (UniqueName: \"kubernetes.io/projected/f9298d17-2ffe-45c3-b494-4cbfa50e98e9-kube-api-access-7mh76\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:48:29.166768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.166739 5688 generic.go:356] "Generic (PLEG): container finished" podID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerID="fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62" exitCode=0 Oct 31 18:48:29.166944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.166804 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" Oct 31 18:48:29.166944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.166820 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" event={"ID":"f9298d17-2ffe-45c3-b494-4cbfa50e98e9","Type":"ContainerDied","Data":"fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62"} Oct 31 18:48:29.166944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.166905 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5" event={"ID":"f9298d17-2ffe-45c3-b494-4cbfa50e98e9","Type":"ContainerDied","Data":"26396e958ee9ecda00616075f73a898dcd19e5f650a1ecd3981cce5a463f604a"} Oct 31 18:48:29.166944 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.166928 5688 scope.go:122] "RemoveContainer" containerID="fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62" Oct 31 18:48:29.177144 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.177126 5688 scope.go:122] "RemoveContainer" containerID="6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e" Oct 31 18:48:29.186257 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.186242 5688 scope.go:122] "RemoveContainer" containerID="fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62" Oct 31 18:48:29.186499 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:29.186476 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62\": container with ID starting with fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62 not found: ID does not exist" containerID="fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62" Oct 31 18:48:29.186586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.186504 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62"} err="failed to get container status \"fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62\": rpc error: code = NotFound desc = could not find container \"fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62\": container with ID starting with fdf7cd03dbbbd7b653bf44a8bece2e8df36aa9b2c31dd156adc963e6fa593f62 not found: ID does not exist" Oct 31 18:48:29.186586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.186521 5688 scope.go:122] "RemoveContainer" containerID="6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e" Oct 31 18:48:29.186759 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:29.186745 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e\": container with ID starting with 6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e not found: ID does not exist" containerID="6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e" Oct 31 18:48:29.186796 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.186764 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e"} err="failed to get container status \"6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e\": rpc error: code = NotFound desc = could not find container \"6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e\": container with ID starting with 6c560a2d94f067718dab794cd4472e87ff6f8d9679ab3a5623bb00b1e546120e not found: ID does not exist" Oct 31 18:48:29.336134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.332544 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5"] Oct 31 18:48:29.336134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.336041 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cd7d17-openshift-default-77b447d7ff-9c5l5"] Oct 31 18:48:29.565750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:29.565721 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" path="/var/lib/kubelet/pods/f9298d17-2ffe-45c3-b494-4cbfa50e98e9/volumes" Oct 31 18:48:32.751561 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.751528 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w"] Oct 31 18:48:32.752139 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752120 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" Oct 31 18:48:32.752239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752141 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" Oct 31 18:48:32.752239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752157 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" Oct 31 18:48:32.752239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752161 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" Oct 31 18:48:32.752354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752300 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" Oct 31 18:48:32.752354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752310 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1c2252-f6c0-40e3-8a99-9b20696e0c5c" containerName="istio-proxy" Oct 31 18:48:32.752354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.752321 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" Oct 31 18:48:32.757096 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.757080 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.800896 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.800871 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5"] Oct 31 18:48:32.801347 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.801334 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" Oct 31 18:48:32.801386 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.801348 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9298d17-2ffe-45c3-b494-4cbfa50e98e9" containerName="istio-proxy" Oct 31 18:48:32.804631 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.804615 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.838911 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.838887 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr5s\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.838927 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839001 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839101 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839053 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839101 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839096 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839217 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839265 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839355 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.839453 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.839438 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.859346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.857376 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w"] Oct 31 18:48:32.905985 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.905953 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5"] Oct 31 18:48:32.940459 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940422 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.940617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940461 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.940617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940483 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.940617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940505 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.940617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940520 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.940617 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940541 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.940888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940620 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.940888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940764 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.940888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940811 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.940888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940846 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940920 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940938 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.940990 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941000 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr5s\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941053 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941084 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.941324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941124 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941145 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.941324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941214 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.941324 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941316 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941338 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.941495 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941343 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jh6\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:32.941967 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.941946 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.943294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.943270 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.943383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.943283 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.997332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.997298 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:32.997469 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:32.997451 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr5s\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:33.042502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042438 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042475 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042509 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042561 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jh6\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042635 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042746 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042681 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042746 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042727 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042778 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042817 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042825 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.042851 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.042776 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.043116 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.043026 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.043291 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.043121 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.043716 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.043695 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.044986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.044962 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.045401 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.045385 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.071957 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.071935 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:33.100507 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.100483 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.100628 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.100605 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jh6\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.118369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.118342 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:33.499284 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.499261 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w"] Oct 31 18:48:33.500831 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:48:33.500805 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03474f2_6403_4af9_b4af_892130cd973c.slice/crio-23d906031f94c3da94cf819b6baec3c97d22959c49ab46c047ad78ff371d2156 WatchSource:0}: Error finding container 23d906031f94c3da94cf819b6baec3c97d22959c49ab46c047ad78ff371d2156: Status 404 returned error can't find the container with id 23d906031f94c3da94cf819b6baec3c97d22959c49ab46c047ad78ff371d2156 Oct 31 18:48:33.503384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.503352 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:33.503456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.503417 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:33.503456 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.503444 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:33.554367 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.554342 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5"] Oct 31 18:48:33.555906 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:48:33.555881 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8049c32c_c2fc_43c5_9ffc_48a62150dd9d.slice/crio-7fc2adc664a3133c03944e4f95664eb088073bcc17e6b6f1dd90352431fc04fc WatchSource:0}: Error finding container 7fc2adc664a3133c03944e4f95664eb088073bcc17e6b6f1dd90352431fc04fc: Status 404 returned error can't find the container with id 7fc2adc664a3133c03944e4f95664eb088073bcc17e6b6f1dd90352431fc04fc Oct 31 18:48:33.558389 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.558359 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:33.558464 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.558425 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:33.558464 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:33.558454 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:48:34.208747 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:34.208711 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" event={"ID":"8049c32c-c2fc-43c5-9ffc-48a62150dd9d","Type":"ContainerStarted","Data":"9f5707a28d8c30f876910043089876dcc18fd1b247d56ec277de3c7a95c1dc16"} Oct 31 18:48:34.209353 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:34.209321 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" event={"ID":"8049c32c-c2fc-43c5-9ffc-48a62150dd9d","Type":"ContainerStarted","Data":"7fc2adc664a3133c03944e4f95664eb088073bcc17e6b6f1dd90352431fc04fc"} Oct 31 18:48:34.210839 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:34.210814 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" event={"ID":"a03474f2-6403-4af9-b4af-892130cd973c","Type":"ContainerStarted","Data":"3050d4514508e27853f4bb69d794c4a5430f6479eaa0b08b8022129b555c3537"} Oct 31 18:48:34.210948 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:34.210855 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" event={"ID":"a03474f2-6403-4af9-b4af-892130cd973c","Type":"ContainerStarted","Data":"23d906031f94c3da94cf819b6baec3c97d22959c49ab46c047ad78ff371d2156"} Oct 31 18:48:34.324125 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:34.324066 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podStartSLOduration=2.324051905 podStartE2EDuration="2.324051905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:34.321464831 +0000 UTC m=+1634.923605974" watchObservedRunningTime="2026-10-31 18:48:34.324051905 +0000 UTC m=+1634.926193048" Oct 31 18:48:35.074587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.074544 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:35.077002 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.076976 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 18:48:35.077295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.077261 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 18:48:35.151276 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.138564 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:35.217214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.150333 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 18:48:35.217214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.209350 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 18:48:35.404290 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.378008 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podStartSLOduration=3.377987518 podStartE2EDuration="3.377987518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:34.454301153 +0000 UTC m=+1635.056442298" watchObservedRunningTime="2026-10-31 18:48:35.377987518 +0000 UTC m=+1635.980128664" Oct 31 18:48:35.405296 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.405253 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265"] Oct 31 18:48:35.425210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.423077 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:35.466224 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.465376 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265"] Oct 31 18:48:35.483221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.480876 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dc7b6-dockercfg-4fmks\"" Oct 31 18:48:35.483221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.480969 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dc7b6\"" Oct 31 18:48:35.535328 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.535282 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j8m\" (UniqueName: \"kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:35.617644 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.535598 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:35.754220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.749066 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j8m\" (UniqueName: \"kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:35.754528 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.754499 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:35.770214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.755643 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:35.884305 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:35.884273 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j8m\" (UniqueName: \"kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:36.073258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:36.072391 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:36.073624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:36.073600 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 18:48:36.073821 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:36.073795 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 18:48:36.118952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:36.118924 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 18:48:36.119064 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:36.118972 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 18:48:36.501440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:36.501402 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265"] Oct 31 18:48:37.075207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.074678 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 18:48:37.075207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.074766 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 18:48:37.118863 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.118828 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 18:48:37.119029 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.118879 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 18:48:37.383364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.383264 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" event={"ID":"12b150a4-f348-4dd0-817a-bfea146134f0","Type":"ContainerStarted","Data":"55817c1cf21e25f78119247d3a591eb0d5d18dd7ec992517b3a64f076eb22ad0"} Oct 31 18:48:37.383364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.383332 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" event={"ID":"12b150a4-f348-4dd0-817a-bfea146134f0","Type":"ContainerStarted","Data":"a926243b915df91db2b61cd9b5c20c5bd59ce414f932dcb93c0d60112058a698"} Oct 31 18:48:37.383762 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.383744 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:37.499952 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:37.499892 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" podStartSLOduration=2.499879777 podStartE2EDuration="2.499879777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:37.498608273 +0000 UTC m=+1638.100749417" watchObservedRunningTime="2026-10-31 18:48:37.499879777 +0000 UTC m=+1638.102020919" Oct 31 18:48:38.079334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:38.075607 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 18:48:38.079334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:38.075669 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 18:48:38.118635 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:38.118608 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 18:48:38.118787 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:38.118662 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 18:48:39.077208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:39.076981 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 18:48:39.077208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:39.077045 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 18:48:39.119265 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:39.119231 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 18:48:39.119629 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:39.119285 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 18:48:40.072469 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:40.072432 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 18:48:40.072627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:40.072494 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 18:48:40.322775 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:40.322694 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:40.324058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:40.324031 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:40.325023 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:40.324992 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:48:41.076256 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:41.076226 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:41.076516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:41.076498 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:41.077259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:41.077242 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:48:48.393172 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:48.393145 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:48:54.904075 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:48:54.904032 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:48:56.817810 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.817781 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj"] Oct 31 18:48:56.821935 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.821912 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:56.835146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.835104 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7zn\" (UniqueName: \"kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:56.835346 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.835328 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:56.880596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.880568 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj"] Oct 31 18:48:56.926217 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.926174 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a4923\"" Oct 31 18:48:56.926354 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.926276 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a4923-dockercfg-4j9vs\"" Oct 31 18:48:56.937009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.936961 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7zn\" (UniqueName: \"kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:56.937327 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.937301 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:56.938856 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:56.938829 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:57.023212 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:57.019630 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7zn\" (UniqueName: \"kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:57.136596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:57.136523 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:57.509542 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:57.509515 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj"] Oct 31 18:48:57.511552 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:48:57.511522 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbba179_5c67_4557_b081_2f850d5d1dd0.slice/crio-f79417ea56d4bbabd8984fbbe151746dae30b94db2f2b69bb756e680f252a727 WatchSource:0}: Error finding container f79417ea56d4bbabd8984fbbe151746dae30b94db2f2b69bb756e680f252a727: Status 404 returned error can't find the container with id f79417ea56d4bbabd8984fbbe151746dae30b94db2f2b69bb756e680f252a727 Oct 31 18:48:58.539596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:58.539543 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" event={"ID":"4fbba179-5c67-4557-b081-2f850d5d1dd0","Type":"ContainerStarted","Data":"00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991"} Oct 31 18:48:58.540083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:58.539650 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" event={"ID":"4fbba179-5c67-4557-b081-2f850d5d1dd0","Type":"ContainerStarted","Data":"f79417ea56d4bbabd8984fbbe151746dae30b94db2f2b69bb756e680f252a727"} Oct 31 18:48:58.540809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:58.540786 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:48:58.655633 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:48:58.655585 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" podStartSLOduration=2.6555719030000002 podStartE2EDuration="2.655571903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:58.654279253 +0000 UTC m=+1659.256420396" watchObservedRunningTime="2026-10-31 18:48:58.655571903 +0000 UTC m=+1659.257713047" Oct 31 18:49:10.554319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:10.554287 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:49:11.847747 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:11.847715 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5"] Oct 31 18:49:16.847986 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:16.847943 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" podUID="d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" containerName="payload-pre-processing" containerID="cri-o://0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f" gracePeriod=25 Oct 31 18:49:16.908516 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:49:16.908485 5688 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28aa4a8_8829_4a3c_ae7e_f6e1b190813b.slice/crio-0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28aa4a8_8829_4a3c_ae7e_f6e1b190813b.slice/crio-conmon-0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 18:49:17.121668 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.121643 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:49:17.214370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.214346 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume\") pod \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " Oct 31 18:49:17.214526 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.214420 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv\" (UniqueName: \"kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv\") pod \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\" (UID: \"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b\") " Oct 31 18:49:17.214695 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.214673 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume" pod "d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" (UID: "d28aa4a8-8829-4a3c-ae7e-f6e1b190813b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:49:17.214857 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.214840 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:17.216573 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.216553 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv" pod "d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" (UID: "d28aa4a8-8829-4a3c-ae7e-f6e1b190813b"). InnerVolumeSpecName "kube-api-access-d9ljv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:49:17.316028 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.316001 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ljv\" (UniqueName: \"kubernetes.io/projected/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b-kube-api-access-d9ljv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:17.648098 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.648067 5688 generic.go:356] "Generic (PLEG): container finished" podID="d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" containerID="0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f" exitCode=0 Oct 31 18:49:17.648282 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.648111 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" Oct 31 18:49:17.648282 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.648158 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" event={"ID":"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b","Type":"ContainerDied","Data":"0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f"} Oct 31 18:49:17.648282 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.648230 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5" event={"ID":"d28aa4a8-8829-4a3c-ae7e-f6e1b190813b","Type":"ContainerDied","Data":"bf032514dd48554f2d1ac7b06330adb4111183d1faaf62f2a51450659e254f35"} Oct 31 18:49:17.648282 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.648252 5688 scope.go:122] "RemoveContainer" containerID="0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f" Oct 31 18:49:17.657624 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.657608 5688 scope.go:122] "RemoveContainer" containerID="0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f" Oct 31 18:49:17.657893 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:49:17.657872 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f\": container with ID starting with 0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f not found: ID does not exist" containerID="0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f" Oct 31 18:49:17.657955 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.657904 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f"} err="failed to get container status \"0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f\": rpc error: code = NotFound desc = could not find container \"0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f\": container with ID starting with 0f9e9de3dd62d164d6e22d1274541cfd50115c5dca8da6dd5b0ab10cfbd2a93f not found: ID does not exist" Oct 31 18:49:17.816640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.816610 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5"] Oct 31 18:49:17.866056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:17.865985 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c21e9e-865464fc59-wpnh5"] Oct 31 18:49:19.580475 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:19.580427 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" path="/var/lib/kubelet/pods/d28aa4a8-8829-4a3c-ae7e-f6e1b190813b/volumes" Oct 31 18:49:20.388987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:20.388947 5688 scope.go:122] "RemoveContainer" containerID="2b858a8a8c72ca9682889752b57bc5b6fdfbf2b6ec5cfcc8691460e1c39b23b5" Oct 31 18:49:20.404043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:20.404024 5688 scope.go:122] "RemoveContainer" containerID="74b8289b466ec024d45a10a20c2cda4ce972c704897576f51d66a8ebc86d6733" Oct 31 18:49:24.904915 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:49:24.904876 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:49:25.294620 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:25.294584 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs"] Oct 31 18:49:25.294837 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:25.294802 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" containerID="cri-o://e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478" gracePeriod=30 Oct 31 18:49:26.241675 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:26.241638 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9p8qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:49:26.242314 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:26.242276 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:49:30.553987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.553963 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:49:30.695359 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695285 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695359 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695326 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695406 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695459 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695489 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695521 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695557 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695598 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695655 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket\") pod \"56cafa0b-8ca9-4b39-ba41-4931964674f8\" (UID: \"56cafa0b-8ca9-4b39-ba41-4931964674f8\") " Oct 31 18:49:30.695798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695684 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:49:30.695938 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695911 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:49:30.696007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.695984 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:49:30.696117 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696098 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:49:30.696169 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696137 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:49:30.696319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696269 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.696319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696291 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.696319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696307 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.696463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696320 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.696463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.696334 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56cafa0b-8ca9-4b39-ba41-4931964674f8-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.698222 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.698193 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:49:30.698498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.698475 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:49:30.698590 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.698524 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "kube-api-access-4cfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:49:30.698590 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.698551 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token" pod "56cafa0b-8ca9-4b39-ba41-4931964674f8" (UID: "56cafa0b-8ca9-4b39-ba41-4931964674f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:49:30.728080 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.728058 5688 generic.go:356] "Generic (PLEG): container finished" podID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerID="e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478" exitCode=0 Oct 31 18:49:30.728208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.728101 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" Oct 31 18:49:30.728208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.728142 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" event={"ID":"56cafa0b-8ca9-4b39-ba41-4931964674f8","Type":"ContainerDied","Data":"e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478"} Oct 31 18:49:30.728291 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.728237 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs" event={"ID":"56cafa0b-8ca9-4b39-ba41-4931964674f8","Type":"ContainerDied","Data":"f12e2ca24d3a851aae1f3f64c47948a4dd8cd774b5d953a5ccb392b62dad7ad1"} Oct 31 18:49:30.728330 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.728265 5688 scope.go:122] "RemoveContainer" containerID="e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478" Oct 31 18:49:30.740432 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.740280 5688 scope.go:122] "RemoveContainer" containerID="e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478" Oct 31 18:49:30.740602 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:49:30.740579 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478\": container with ID starting with e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478 not found: ID does not exist" containerID="e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478" Oct 31 18:49:30.740642 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.740614 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478"} err="failed to get container status \"e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478\": rpc error: code = NotFound desc = could not find container \"e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478\": container with ID starting with e304d0d797671f115ca29e3247c7d9873c899af427ed156bf28ff99e32bf6478 not found: ID does not exist" Oct 31 18:49:30.797682 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.797654 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.797682 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.797681 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.797862 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.797689 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56cafa0b-8ca9-4b39-ba41-4931964674f8-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.797862 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.797698 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfb8\" (UniqueName: \"kubernetes.io/projected/56cafa0b-8ca9-4b39-ba41-4931964674f8-kube-api-access-4cfb8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:30.892258 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.892224 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs"] Oct 31 18:49:30.895293 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:30.895268 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9p8qs"] Oct 31 18:49:31.565923 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:31.565889 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" path="/var/lib/kubelet/pods/56cafa0b-8ca9-4b39-ba41-4931964674f8/volumes" Oct 31 18:49:48.460625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:48.460590 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8"] Oct 31 18:49:48.644153 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:48.644101 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97"] Oct 31 18:49:53.461412 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.461372 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" podUID="069e7f65-9c46-4dbf-9465-7bd4bcedf63e" containerName="payload-processing" containerID="cri-o://61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb" gracePeriod=25 Oct 31 18:49:53.645649 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.645565 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" podUID="d87787af-4e3c-4fd1-a3a4-918505116192" containerName="payload-pre-processing" containerID="cri-o://97c3977835dd93b6ee9676052d6ab8971b1138368e5261cd26c59300871e1e05" gracePeriod=25 Oct 31 18:49:53.778122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.778094 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:49:53.900030 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.900001 5688 generic.go:356] "Generic (PLEG): container finished" podID="069e7f65-9c46-4dbf-9465-7bd4bcedf63e" containerID="61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb" exitCode=0 Oct 31 18:49:53.900162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.900048 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" Oct 31 18:49:53.900162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.900074 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" event={"ID":"069e7f65-9c46-4dbf-9465-7bd4bcedf63e","Type":"ContainerDied","Data":"61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb"} Oct 31 18:49:53.900162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.900137 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8" event={"ID":"069e7f65-9c46-4dbf-9465-7bd4bcedf63e","Type":"ContainerDied","Data":"cab5352170c11cb7040e138ecde9ddd68272335a9ae3c39567973f75faaa75fe"} Oct 31 18:49:53.900162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.900163 5688 scope.go:122] "RemoveContainer" containerID="61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb" Oct 31 18:49:53.902496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.902465 5688 generic.go:356] "Generic (PLEG): container finished" podID="d87787af-4e3c-4fd1-a3a4-918505116192" containerID="97c3977835dd93b6ee9676052d6ab8971b1138368e5261cd26c59300871e1e05" exitCode=0 Oct 31 18:49:53.902590 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.902526 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" event={"ID":"d87787af-4e3c-4fd1-a3a4-918505116192","Type":"ContainerDied","Data":"97c3977835dd93b6ee9676052d6ab8971b1138368e5261cd26c59300871e1e05"} Oct 31 18:49:53.904085 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.904064 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:49:53.909713 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.909635 5688 scope.go:122] "RemoveContainer" containerID="61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb" Oct 31 18:49:53.909909 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:49:53.909889 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb\": container with ID starting with 61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb not found: ID does not exist" containerID="61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb" Oct 31 18:49:53.909967 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.909917 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb"} err="failed to get container status \"61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb\": rpc error: code = NotFound desc = could not find container \"61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb\": container with ID starting with 61f466df85eaab562f3000206399c21b73691b7ba418a3da68ff16e43c9e9dcb not found: ID does not exist" Oct 31 18:49:53.943498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.943457 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx\" (UniqueName: \"kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx\") pod \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " Oct 31 18:49:53.943620 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.943529 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume\") pod \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\" (UID: \"069e7f65-9c46-4dbf-9465-7bd4bcedf63e\") " Oct 31 18:49:53.943894 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.943872 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume" pod "069e7f65-9c46-4dbf-9465-7bd4bcedf63e" (UID: "069e7f65-9c46-4dbf-9465-7bd4bcedf63e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:49:53.945517 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:53.945497 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx" pod "069e7f65-9c46-4dbf-9465-7bd4bcedf63e" (UID: "069e7f65-9c46-4dbf-9465-7bd4bcedf63e"). InnerVolumeSpecName "kube-api-access-vnfrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:49:54.044954 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.044928 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5\" (UniqueName: \"kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5\") pod \"d87787af-4e3c-4fd1-a3a4-918505116192\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " Oct 31 18:49:54.045095 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.045010 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume\") pod \"d87787af-4e3c-4fd1-a3a4-918505116192\" (UID: \"d87787af-4e3c-4fd1-a3a4-918505116192\") " Oct 31 18:49:54.045321 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.045300 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume" pod "d87787af-4e3c-4fd1-a3a4-918505116192" (UID: "d87787af-4e3c-4fd1-a3a4-918505116192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:49:54.045516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.045490 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfrx\" (UniqueName: \"kubernetes.io/projected/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-kube-api-access-vnfrx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:54.045918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.045519 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069e7f65-9c46-4dbf-9465-7bd4bcedf63e-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:54.045918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.045535 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87787af-4e3c-4fd1-a3a4-918505116192-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:54.047162 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.047135 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5" pod "d87787af-4e3c-4fd1-a3a4-918505116192" (UID: "d87787af-4e3c-4fd1-a3a4-918505116192"). InnerVolumeSpecName "kube-api-access-55cg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:49:54.146543 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.146512 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55cg5\" (UniqueName: \"kubernetes.io/projected/d87787af-4e3c-4fd1-a3a4-918505116192-kube-api-access-55cg5\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:49:54.321057 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.320987 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8"] Oct 31 18:49:54.324224 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.324162 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66032c-687ccdf449-94wt8"] Oct 31 18:49:54.905672 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:49:54.905637 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:49:54.910523 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.910502 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" Oct 31 18:49:54.910647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.910501 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97" event={"ID":"d87787af-4e3c-4fd1-a3a4-918505116192","Type":"ContainerDied","Data":"721e90b6f7c67e998d08d95fe96d0b0888066a2a5418239480ce207aae7c0e5b"} Oct 31 18:49:54.910647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:54.910619 5688 scope.go:122] "RemoveContainer" containerID="97c3977835dd93b6ee9676052d6ab8971b1138368e5261cd26c59300871e1e05" Oct 31 18:49:55.089504 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:55.089478 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97"] Oct 31 18:49:55.138918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:55.138889 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66032c-5f4fd754ff-r8p97"] Oct 31 18:49:55.575405 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:55.575349 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069e7f65-9c46-4dbf-9465-7bd4bcedf63e" path="/var/lib/kubelet/pods/069e7f65-9c46-4dbf-9465-7bd4bcedf63e/volumes" Oct 31 18:49:55.576448 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:49:55.576425 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87787af-4e3c-4fd1-a3a4-918505116192" path="/var/lib/kubelet/pods/d87787af-4e3c-4fd1-a3a4-918505116192/volumes" Oct 31 18:50:06.722786 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:06.722752 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf"] Oct 31 18:50:06.723277 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:06.722991 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" containerID="cri-o://4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400" gracePeriod=30 Oct 31 18:50:06.735937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:06.735912 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w"] Oct 31 18:50:06.736158 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:06.736120 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" containerID="cri-o://3050d4514508e27853f4bb69d794c4a5430f6479eaa0b08b8022129b555c3537" gracePeriod=30 Oct 31 18:50:06.738060 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:06.738040 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5"] Oct 31 18:50:06.738244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:06.738208 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" containerID="cri-o://9f5707a28d8c30f876910043089876dcc18fd1b247d56ec277de3c7a95c1dc16" gracePeriod=30 Oct 31 18:50:10.177246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:10.177212 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:50:10.177640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:10.177264 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:50:10.326852 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:10.326819 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:50:10.327033 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:10.326873 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:50:11.079055 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:11.079022 5688 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:50:11.079244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:11.079076 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:50:12.006608 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.006574 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:50:12.027662 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.027631 5688 generic.go:356] "Generic (PLEG): container finished" podID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerID="9f5707a28d8c30f876910043089876dcc18fd1b247d56ec277de3c7a95c1dc16" exitCode=0 Oct 31 18:50:12.027824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.027714 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" event={"ID":"8049c32c-c2fc-43c5-9ffc-48a62150dd9d","Type":"ContainerDied","Data":"9f5707a28d8c30f876910043089876dcc18fd1b247d56ec277de3c7a95c1dc16"} Oct 31 18:50:12.029462 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.029441 5688 generic.go:356] "Generic (PLEG): container finished" podID="a03474f2-6403-4af9-b4af-892130cd973c" containerID="3050d4514508e27853f4bb69d794c4a5430f6479eaa0b08b8022129b555c3537" exitCode=0 Oct 31 18:50:12.029582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.029519 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" event={"ID":"a03474f2-6403-4af9-b4af-892130cd973c","Type":"ContainerDied","Data":"3050d4514508e27853f4bb69d794c4a5430f6479eaa0b08b8022129b555c3537"} Oct 31 18:50:12.030908 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.030886 5688 generic.go:356] "Generic (PLEG): container finished" podID="130d7162-286d-4211-9720-67cb863812e3" containerID="4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400" exitCode=0 Oct 31 18:50:12.031017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.030925 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" event={"ID":"130d7162-286d-4211-9720-67cb863812e3","Type":"ContainerDied","Data":"4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400"} Oct 31 18:50:12.031017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.030953 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" Oct 31 18:50:12.031017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.030970 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf" event={"ID":"130d7162-286d-4211-9720-67cb863812e3","Type":"ContainerDied","Data":"bf5092f11484e619f175745905cf10bef853939dde326ebf3725e80addc2eba8"} Oct 31 18:50:12.031017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.030998 5688 scope.go:122] "RemoveContainer" containerID="4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400" Oct 31 18:50:12.041512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.041493 5688 scope.go:122] "RemoveContainer" containerID="4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400" Oct 31 18:50:12.041804 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:50:12.041781 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400\": container with ID starting with 4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400 not found: ID does not exist" containerID="4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400" Oct 31 18:50:12.041934 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.041830 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400"} err="failed to get container status \"4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400\": rpc error: code = NotFound desc = could not find container \"4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400\": container with ID starting with 4fc9e5c6da36e7fe38cac7c434aadce15f50784c31b31c26091c4f0a8af9e400 not found: ID does not exist" Oct 31 18:50:12.075582 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.075563 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:50:12.077595 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.077571 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:50:12.106623 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106598 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.106710 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106634 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.106760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106703 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.106760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106732 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.106866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106761 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.106866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106791 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.106866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106826 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.106866 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106860 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.107056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106868 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.107056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106885 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.107056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106912 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.107056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106967 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.107056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106985 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.107056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.106998 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107070 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107103 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107125 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107166 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107230 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107266 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107302 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107284 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107329 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107340 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107377 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:50:12.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107391 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107461 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107492 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107530 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107540 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107591 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107665 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107678 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107723 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token\") pod \"130d7162-286d-4211-9720-67cb863812e3\" (UID: \"130d7162-286d-4211-9720-67cb863812e3\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107753 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy\") pod \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\" (UID: \"8049c32c-c2fc-43c5-9ffc-48a62150dd9d\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107784 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket\") pod \"a03474f2-6403-4af9-b4af-892130cd973c\" (UID: \"a03474f2-6403-4af9-b4af-892130cd973c\") " Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.107790 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:50:12.108126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108011 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108480 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108504 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108518 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130d7162-286d-4211-9720-67cb863812e3-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108531 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108544 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108556 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108569 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108581 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108593 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108605 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.108768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.108618 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03474f2-6403-4af9-b4af-892130cd973c-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.109838 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.109810 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:50:12.110312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.110288 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.110411 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.110346 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.110478 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.110406 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:50:12.110721 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.110700 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.110919 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.110899 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.110990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.110937 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.111199 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.111149 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:50:12.111604 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.111579 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "kube-api-access-plr5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:12.111712 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.111625 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "kube-api-access-kbgn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:12.111778 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.111705 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token" pod "a03474f2-6403-4af9-b4af-892130cd973c" (UID: "a03474f2-6403-4af9-b4af-892130cd973c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:12.111909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.111891 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "kube-api-access-t7jh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:12.112140 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.112106 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.112394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.112373 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:50:12.112975 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.112958 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token" pod "8049c32c-c2fc-43c5-9ffc-48a62150dd9d" (UID: "8049c32c-c2fc-43c5-9ffc-48a62150dd9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:12.113405 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.113390 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token" pod "130d7162-286d-4211-9720-67cb863812e3" (UID: "130d7162-286d-4211-9720-67cb863812e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:12.209768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209731 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plr5s\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-kube-api-access-plr5s\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.209768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209773 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209788 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209801 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209814 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03474f2-6403-4af9-b4af-892130cd973c-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209828 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209841 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209854 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7jh6\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-kube-api-access-t7jh6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209867 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209881 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03474f2-6403-4af9-b4af-892130cd973c-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209896 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130d7162-286d-4211-9720-67cb863812e3-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209909 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03474f2-6403-4af9-b4af-892130cd973c-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209922 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209935 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbgn7\" (UniqueName: \"kubernetes.io/projected/130d7162-286d-4211-9720-67cb863812e3-kube-api-access-kbgn7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209950 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8049c32c-c2fc-43c5-9ffc-48a62150dd9d-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.210007 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.209964 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130d7162-286d-4211-9720-67cb863812e3-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:12.578003 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.577969 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf"] Oct 31 18:50:12.588695 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:12.588667 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-sfxtf"] Oct 31 18:50:13.038114 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.038037 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" Oct 31 18:50:13.038114 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.038055 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5" event={"ID":"8049c32c-c2fc-43c5-9ffc-48a62150dd9d","Type":"ContainerDied","Data":"7fc2adc664a3133c03944e4f95664eb088073bcc17e6b6f1dd90352431fc04fc"} Oct 31 18:50:13.038611 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.038114 5688 scope.go:122] "RemoveContainer" containerID="9f5707a28d8c30f876910043089876dcc18fd1b247d56ec277de3c7a95c1dc16" Oct 31 18:50:13.040238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.039916 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" Oct 31 18:50:13.040238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.039923 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w" event={"ID":"a03474f2-6403-4af9-b4af-892130cd973c","Type":"ContainerDied","Data":"23d906031f94c3da94cf819b6baec3c97d22959c49ab46c047ad78ff371d2156"} Oct 31 18:50:13.049165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.049144 5688 scope.go:122] "RemoveContainer" containerID="3050d4514508e27853f4bb69d794c4a5430f6479eaa0b08b8022129b555c3537" Oct 31 18:50:13.201388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.201352 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5"] Oct 31 18:50:13.254243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.254205 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-w9dj5"] Oct 31 18:50:13.364530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.364449 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w"] Oct 31 18:50:13.364530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.364505 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-j2q4w"] Oct 31 18:50:13.565593 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.565564 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130d7162-286d-4211-9720-67cb863812e3" path="/var/lib/kubelet/pods/130d7162-286d-4211-9720-67cb863812e3/volumes" Oct 31 18:50:13.566003 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.565989 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" path="/var/lib/kubelet/pods/8049c32c-c2fc-43c5-9ffc-48a62150dd9d/volumes" Oct 31 18:50:13.566407 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:13.566394 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03474f2-6403-4af9-b4af-892130cd973c" path="/var/lib/kubelet/pods/a03474f2-6403-4af9-b4af-892130cd973c/volumes" Oct 31 18:50:22.227320 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227284 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv"] Oct 31 18:50:22.227912 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227892 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" containerName="payload-pre-processing" Oct 31 18:50:22.227987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227914 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" containerName="payload-pre-processing" Oct 31 18:50:22.227987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227939 5688 cpu_manager.go:414] "containerMap: removing container" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" Oct 31 18:50:22.227987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227946 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" Oct 31 18:50:22.227987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227958 5688 cpu_manager.go:414] "containerMap: removing container" podUID="069e7f65-9c46-4dbf-9465-7bd4bcedf63e" containerName="payload-processing" Oct 31 18:50:22.227987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227965 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069e7f65-9c46-4dbf-9465-7bd4bcedf63e" containerName="payload-processing" Oct 31 18:50:22.227987 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227986 5688 cpu_manager.go:414] "containerMap: removing container" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.227993 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.228022 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d87787af-4e3c-4fd1-a3a4-918505116192" containerName="payload-pre-processing" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.228029 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87787af-4e3c-4fd1-a3a4-918505116192" containerName="payload-pre-processing" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.228052 5688 cpu_manager.go:414] "containerMap: removing container" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.228059 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.228071 5688 cpu_manager.go:414] "containerMap: removing container" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" Oct 31 18:50:22.228310 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.228078 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" Oct 31 18:50:22.229264 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229240 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28aa4a8-8829-4a3c-ae7e-f6e1b190813b" containerName="payload-pre-processing" Oct 31 18:50:22.229404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229270 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cafa0b-8ca9-4b39-ba41-4931964674f8" containerName="istio-proxy" Oct 31 18:50:22.229404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229301 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="069e7f65-9c46-4dbf-9465-7bd4bcedf63e" containerName="payload-processing" Oct 31 18:50:22.229404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229317 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87787af-4e3c-4fd1-a3a4-918505116192" containerName="payload-pre-processing" Oct 31 18:50:22.229404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229332 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03474f2-6403-4af9-b4af-892130cd973c" containerName="istio-proxy" Oct 31 18:50:22.229404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229343 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="130d7162-286d-4211-9720-67cb863812e3" containerName="istio-proxy" Oct 31 18:50:22.229404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.229350 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="8049c32c-c2fc-43c5-9ffc-48a62150dd9d" containerName="istio-proxy" Oct 31 18:50:22.234826 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.234807 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.321250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.307228 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv"] Oct 31 18:50:22.321250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.312929 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk6sk\" (UniqueName: \"kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk\") pod \"payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.321250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.313093 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.374208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.373017 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-aa4693\"" Oct 31 18:50:22.374208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.373774 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-aa4693-dockercfg-28km8\"" Oct 31 18:50:22.425627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.425586 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.425785 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.425706 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk6sk\" (UniqueName: \"kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk\") pod \"payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.426846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.426817 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.482915 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.482890 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk6sk\" (UniqueName: \"kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk\") pod \"payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.547609 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.547583 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:22.923664 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.923479 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv"] Oct 31 18:50:22.926461 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:50:22.926431 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f36eb15_61f5_42b4_9ac4_f6630200d37a.slice/crio-94ba7b5b57b1a28336cbda6446a2b9eb5eebd86978fdcc55dc377554eb0f0e09 WatchSource:0}: Error finding container 94ba7b5b57b1a28336cbda6446a2b9eb5eebd86978fdcc55dc377554eb0f0e09: Status 404 returned error can't find the container with id 94ba7b5b57b1a28336cbda6446a2b9eb5eebd86978fdcc55dc377554eb0f0e09 Oct 31 18:50:22.987396 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:22.987250 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265"] Oct 31 18:50:23.122083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:23.122044 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" event={"ID":"0f36eb15-61f5-42b4-9ac4-f6630200d37a","Type":"ContainerStarted","Data":"9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105"} Oct 31 18:50:23.122246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:23.122104 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" event={"ID":"0f36eb15-61f5-42b4-9ac4-f6630200d37a","Type":"ContainerStarted","Data":"94ba7b5b57b1a28336cbda6446a2b9eb5eebd86978fdcc55dc377554eb0f0e09"} Oct 31 18:50:23.122246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:23.122124 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:23.235788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:23.235733 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" podStartSLOduration=1.235716798 podStartE2EDuration="1.235716798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:23.234769799 +0000 UTC m=+1743.836911123" watchObservedRunningTime="2026-10-31 18:50:23.235716798 +0000 UTC m=+1743.837857943" Oct 31 18:50:24.904618 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:50:24.904588 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:50:27.988613 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:27.988568 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" podUID="12b150a4-f348-4dd0-817a-bfea146134f0" containerName="payload-pre-processing" containerID="cri-o://55817c1cf21e25f78119247d3a591eb0d5d18dd7ec992517b3a64f076eb22ad0" gracePeriod=25 Oct 31 18:50:28.265807 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.265726 5688 generic.go:356] "Generic (PLEG): container finished" podID="12b150a4-f348-4dd0-817a-bfea146134f0" containerID="55817c1cf21e25f78119247d3a591eb0d5d18dd7ec992517b3a64f076eb22ad0" exitCode=0 Oct 31 18:50:28.265949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.265819 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" event={"ID":"12b150a4-f348-4dd0-817a-bfea146134f0","Type":"ContainerDied","Data":"55817c1cf21e25f78119247d3a591eb0d5d18dd7ec992517b3a64f076eb22ad0"} Oct 31 18:50:28.447361 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.447334 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:50:28.540953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.540868 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m\" (UniqueName: \"kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m\") pod \"12b150a4-f348-4dd0-817a-bfea146134f0\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " Oct 31 18:50:28.540953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.540945 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume\") pod \"12b150a4-f348-4dd0-817a-bfea146134f0\" (UID: \"12b150a4-f348-4dd0-817a-bfea146134f0\") " Oct 31 18:50:28.541417 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.541389 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume" pod "12b150a4-f348-4dd0-817a-bfea146134f0" (UID: "12b150a4-f348-4dd0-817a-bfea146134f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:50:28.543150 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.543122 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m" pod "12b150a4-f348-4dd0-817a-bfea146134f0" (UID: "12b150a4-f348-4dd0-817a-bfea146134f0"). InnerVolumeSpecName "kube-api-access-s7j8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:28.642719 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.642691 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7j8m\" (UniqueName: \"kubernetes.io/projected/12b150a4-f348-4dd0-817a-bfea146134f0-kube-api-access-s7j8m\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:28.642719 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:28.642718 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b150a4-f348-4dd0-817a-bfea146134f0-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:29.282047 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.282016 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" Oct 31 18:50:29.287792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.282016 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" event={"ID":"12b150a4-f348-4dd0-817a-bfea146134f0","Type":"ContainerDied","Data":"a926243b915df91db2b61cd9b5c20c5bd59ce414f932dcb93c0d60112058a698"} Oct 31 18:50:29.287792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.282375 5688 scope.go:122] "RemoveContainer" containerID="55817c1cf21e25f78119247d3a591eb0d5d18dd7ec992517b3a64f076eb22ad0" Oct 31 18:50:29.392077 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.391269 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265" podUID="12b150a4-f348-4dd0-817a-bfea146134f0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.106:50052\" within 1s: context deadline exceeded" Oct 31 18:50:29.392077 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:50:29.391629 5688 logging.go:55] [core] [Channel #278 SubChannel #279]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.106:50052", ServerName: "10.132.0.106:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.106:50052: operation was canceled" Oct 31 18:50:29.452519 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.452354 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265"] Oct 31 18:50:29.501240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.501206 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-785799c4c-wm265"] Oct 31 18:50:29.503053 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.503033 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt"] Oct 31 18:50:29.503516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.503503 5688 cpu_manager.go:414] "containerMap: removing container" podUID="12b150a4-f348-4dd0-817a-bfea146134f0" containerName="payload-pre-processing" Oct 31 18:50:29.503562 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.503517 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b150a4-f348-4dd0-817a-bfea146134f0" containerName="payload-pre-processing" Oct 31 18:50:29.503607 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.503596 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b150a4-f348-4dd0-817a-bfea146134f0" containerName="payload-pre-processing" Oct 31 18:50:29.508087 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.508072 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.553747 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.553686 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sg7g\" (UniqueName: \"kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g\") pod \"payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.553847 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.553809 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.561894 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.560132 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-100faf-dockercfg-mx5n2\"" Oct 31 18:50:29.561894 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.560164 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-100faf\"" Oct 31 18:50:29.567549 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.567523 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b150a4-f348-4dd0-817a-bfea146134f0" path="/var/lib/kubelet/pods/12b150a4-f348-4dd0-817a-bfea146134f0/volumes" Oct 31 18:50:29.567926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.567907 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt"] Oct 31 18:50:29.690210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.688276 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.690210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.688422 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sg7g\" (UniqueName: \"kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g\") pod \"payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.690210 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.689575 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.717318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.717262 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr"] Oct 31 18:50:29.725778 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.725746 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr"] Oct 31 18:50:29.725949 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.725860 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:29.753366 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.753341 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sg7g\" (UniqueName: \"kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g\") pod \"payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.789926 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.789901 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblbx\" (UniqueName: \"kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx\") pod \"payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:29.790058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.789933 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:29.821511 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.821453 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:29.893269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.891981 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblbx\" (UniqueName: \"kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx\") pod \"payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:29.893269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.892034 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:29.893269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.892943 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:29.949867 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:29.949842 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblbx\" (UniqueName: \"kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx\") pod \"payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:30.058323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.058297 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:30.197408 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.197386 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt"] Oct 31 18:50:30.200295 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:50:30.200250 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9a79be_60c5_4073_a085_a198000c6234.slice/crio-1cd6b7785ae621b382cb24956853aeeefa5856ead0064a184709cf478c82b82b WatchSource:0}: Error finding container 1cd6b7785ae621b382cb24956853aeeefa5856ead0064a184709cf478c82b82b: Status 404 returned error can't find the container with id 1cd6b7785ae621b382cb24956853aeeefa5856ead0064a184709cf478c82b82b Oct 31 18:50:30.289033 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.289003 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" event={"ID":"4c9a79be-60c5-4073-a085-a198000c6234","Type":"ContainerStarted","Data":"ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098"} Oct 31 18:50:30.289436 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.289081 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" event={"ID":"4c9a79be-60c5-4073-a085-a198000c6234","Type":"ContainerStarted","Data":"1cd6b7785ae621b382cb24956853aeeefa5856ead0064a184709cf478c82b82b"} Oct 31 18:50:30.289436 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.289226 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:30.401382 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.401299 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" podStartSLOduration=1.401286688 podStartE2EDuration="1.401286688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:30.400750014 +0000 UTC m=+1751.002891150" watchObservedRunningTime="2026-10-31 18:50:30.401286688 +0000 UTC m=+1751.003427832" Oct 31 18:50:30.431521 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:30.431497 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr"] Oct 31 18:50:30.433834 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:50:30.433806 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9ca062_3f78_44b2_b90f_fd265dc5b2f3.slice/crio-7fc8b62f6bb23efda5323eacb52b7cea2d4b8d31126d4a766e8ddd3f61634f47 WatchSource:0}: Error finding container 7fc8b62f6bb23efda5323eacb52b7cea2d4b8d31126d4a766e8ddd3f61634f47: Status 404 returned error can't find the container with id 7fc8b62f6bb23efda5323eacb52b7cea2d4b8d31126d4a766e8ddd3f61634f47 Oct 31 18:50:31.296798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:31.296766 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" event={"ID":"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3","Type":"ContainerStarted","Data":"a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7"} Oct 31 18:50:31.297243 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:31.296818 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" event={"ID":"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3","Type":"ContainerStarted","Data":"7fc8b62f6bb23efda5323eacb52b7cea2d4b8d31126d4a766e8ddd3f61634f47"} Oct 31 18:50:31.409930 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:31.409876 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" podStartSLOduration=2.409861946 podStartE2EDuration="2.409861946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:31.408135019 +0000 UTC m=+1752.010276163" watchObservedRunningTime="2026-10-31 18:50:31.409861946 +0000 UTC m=+1752.012003089" Oct 31 18:50:32.301992 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:32.301959 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:34.148482 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:34.148457 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:50:36.752820 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.751722 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq"] Oct 31 18:50:36.758332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.758302 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:36.807249 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.807219 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq"] Oct 31 18:50:36.854026 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.853989 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-335e78\"" Oct 31 18:50:36.854165 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.854054 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-335e78-dockercfg-7g7hq\"" Oct 31 18:50:36.880963 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.880935 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgp6\" (UniqueName: \"kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6\") pod \"payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:36.881076 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.881003 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:36.982279 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.982252 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:36.982421 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.982365 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgp6\" (UniqueName: \"kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6\") pod \"payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:36.982892 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:36.982874 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:37.038511 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:37.038454 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgp6\" (UniqueName: \"kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6\") pod \"payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:37.073327 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:37.073305 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:37.444662 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:37.444636 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq"] Oct 31 18:50:37.446726 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:50:37.446696 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3a8c90_1fbc_47af_a7cf_11030d7ffea4.slice/crio-abc5fdb84a214e9488e9443971fc517260abe20c7d510f9d3c8e8a6e77e66093 WatchSource:0}: Error finding container abc5fdb84a214e9488e9443971fc517260abe20c7d510f9d3c8e8a6e77e66093: Status 404 returned error can't find the container with id abc5fdb84a214e9488e9443971fc517260abe20c7d510f9d3c8e8a6e77e66093 Oct 31 18:50:38.363512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:38.363480 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" event={"ID":"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4","Type":"ContainerStarted","Data":"db22beffe1f811af9f1d7df15192fd2dd13466fa76ff82a3e7e1986598ef80fc"} Oct 31 18:50:38.363910 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:38.363533 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" event={"ID":"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4","Type":"ContainerStarted","Data":"abc5fdb84a214e9488e9443971fc517260abe20c7d510f9d3c8e8a6e77e66093"} Oct 31 18:50:38.363910 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:38.363611 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:38.479370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:38.479323 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" podStartSLOduration=2.479309997 podStartE2EDuration="2.479309997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:38.477686397 +0000 UTC m=+1759.079827539" watchObservedRunningTime="2026-10-31 18:50:38.479309997 +0000 UTC m=+1759.081451158" Oct 31 18:50:38.578319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:38.578290 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh"] Oct 31 18:50:38.578496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:38.578471 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" podUID="9390acf7-e67b-4306-a393-a1a242e77c46" containerName="maas-api" containerID="cri-o://f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d" gracePeriod=30 Oct 31 18:50:39.092437 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.092138 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:50:39.244261 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.244227 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\") pod \"9390acf7-e67b-4306-a393-a1a242e77c46\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " Oct 31 18:50:39.244444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.244322 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf\") pod \"9390acf7-e67b-4306-a393-a1a242e77c46\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " Oct 31 18:50:39.244444 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.244401 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca\") pod \"9390acf7-e67b-4306-a393-a1a242e77c46\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " Oct 31 18:50:39.244566 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.244533 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs\") pod \"9390acf7-e67b-4306-a393-a1a242e77c46\" (UID: \"9390acf7-e67b-4306-a393-a1a242e77c46\") " Oct 31 18:50:39.244852 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.244821 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca" pod "9390acf7-e67b-4306-a393-a1a242e77c46" (UID: "9390acf7-e67b-4306-a393-a1a242e77c46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:50:39.245117 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.244946 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9390acf7-e67b-4306-a393-a1a242e77c46-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:39.246502 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.246474 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls" pod "9390acf7-e67b-4306-a393-a1a242e77c46" (UID: "9390acf7-e67b-4306-a393-a1a242e77c46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:50:39.247095 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.247066 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs" pod "9390acf7-e67b-4306-a393-a1a242e77c46" (UID: "9390acf7-e67b-4306-a393-a1a242e77c46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:50:39.247228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.247067 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf" pod "9390acf7-e67b-4306-a393-a1a242e77c46" (UID: "9390acf7-e67b-4306-a393-a1a242e77c46"). InnerVolumeSpecName "kube-api-access-nsftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:39.346671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.346595 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:39.346671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.346624 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9390acf7-e67b-4306-a393-a1a242e77c46-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:39.346671 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.346634 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/9390acf7-e67b-4306-a393-a1a242e77c46-kube-api-access-nsftf\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:39.370225 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.370195 5688 generic.go:356] "Generic (PLEG): container finished" podID="9390acf7-e67b-4306-a393-a1a242e77c46" containerID="f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d" exitCode=0 Oct 31 18:50:39.370618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.370258 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" Oct 31 18:50:39.370618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.370283 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" event={"ID":"9390acf7-e67b-4306-a393-a1a242e77c46","Type":"ContainerDied","Data":"f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d"} Oct 31 18:50:39.370618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.370368 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh" event={"ID":"9390acf7-e67b-4306-a393-a1a242e77c46","Type":"ContainerDied","Data":"6c80e990bd0d632e7e18b602fdbfa3df03487db6a7f4774ac8bfb1b9b9479b1a"} Oct 31 18:50:39.370618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.370396 5688 scope.go:122] "RemoveContainer" containerID="f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d" Oct 31 18:50:39.380090 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.380075 5688 scope.go:122] "RemoveContainer" containerID="f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d" Oct 31 18:50:39.380428 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:50:39.380404 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d\": container with ID starting with f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d not found: ID does not exist" containerID="f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d" Oct 31 18:50:39.380537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.380435 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d"} err="failed to get container status \"f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d\": rpc error: code = NotFound desc = could not find container \"f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d\": container with ID starting with f18efed17b7a988f8ffbd9f47c56dce68eb87a121c65c18522616eedd31eb63d not found: ID does not exist" Oct 31 18:50:39.547208 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.546241 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh"] Oct 31 18:50:39.569216 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.562212 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66032c-86bbbc5777-ndjxh"] Oct 31 18:50:39.602158 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:39.602075 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9390acf7-e67b-4306-a393-a1a242e77c46" path="/var/lib/kubelet/pods/9390acf7-e67b-4306-a393-a1a242e77c46/volumes" Oct 31 18:50:41.299809 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:41.299775 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:50:41.909428 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:41.909394 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj"] Oct 31 18:50:42.218209 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.218111 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl"] Oct 31 18:50:42.218888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.218861 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9390acf7-e67b-4306-a393-a1a242e77c46" containerName="maas-api" Oct 31 18:50:42.218888 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.218885 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9390acf7-e67b-4306-a393-a1a242e77c46" containerName="maas-api" Oct 31 18:50:42.219083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.218999 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9390acf7-e67b-4306-a393-a1a242e77c46" containerName="maas-api" Oct 31 18:50:42.223780 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.223760 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.300020 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.299993 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ceef3b-openshift-default-dockercfg-f4g4z\"" Oct 31 18:50:42.301833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.301807 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl"] Oct 31 18:50:42.399169 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399136 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399172 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399269 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399320 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399366 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399453 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399478 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqcb\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399589 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399515 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.399625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.399614 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.505005 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.504962 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.505704 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.505670 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.506692 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.506661 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.507146 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.507130 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.507577 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.507554 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.508442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.508422 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.509012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.508993 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.509451 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.509434 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqcb\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.509891 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.509870 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.510126 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.509375 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.510268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.507507 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.510352 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.507071 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.510429 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.506586 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.510743 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.510724 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.518788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.518764 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.536574 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.536533 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.562458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.562425 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.565600 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.565579 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqcb\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb\") pod \"e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:42.845379 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:42.845303 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:43.304648 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:43.304614 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl"] Oct 31 18:50:43.322390 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:43.322338 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:50:43.327244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:43.322578 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:50:43.327501 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:43.327476 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:50:43.340954 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:43.331310 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:50:43.396749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:43.396723 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" event={"ID":"294c06f0-8bb4-483a-ba49-b99d8182f7db","Type":"ContainerStarted","Data":"41ae0b2671af28fa1294b212b520f501d632359aac5d931c73d6a16dbfbe4f2a"} Oct 31 18:50:44.405344 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:44.405314 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" event={"ID":"294c06f0-8bb4-483a-ba49-b99d8182f7db","Type":"ContainerStarted","Data":"b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a"} Oct 31 18:50:44.846275 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:44.846239 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:44.852238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:44.852205 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:44.968343 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:44.968296 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podStartSLOduration=2.968281776 podStartE2EDuration="2.968281776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:44.621288842 +0000 UTC m=+1765.223429986" watchObservedRunningTime="2026-10-31 18:50:44.968281776 +0000 UTC m=+1765.570423002" Oct 31 18:50:45.410840 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:45.410803 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:45.412022 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:45.411997 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:50:46.910085 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:46.910047 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" podUID="4fbba179-5c67-4557-b081-2f850d5d1dd0" containerName="payload-pre-processing" containerID="cri-o://00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991" gracePeriod=25 Oct 31 18:50:47.167294 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.167237 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:50:47.269312 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.269279 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn\" (UniqueName: \"kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn\") pod \"4fbba179-5c67-4557-b081-2f850d5d1dd0\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " Oct 31 18:50:47.269506 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.269453 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume\") pod \"4fbba179-5c67-4557-b081-2f850d5d1dd0\" (UID: \"4fbba179-5c67-4557-b081-2f850d5d1dd0\") " Oct 31 18:50:47.269886 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.269855 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume" pod "4fbba179-5c67-4557-b081-2f850d5d1dd0" (UID: "4fbba179-5c67-4557-b081-2f850d5d1dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:50:47.271647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.271626 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn" pod "4fbba179-5c67-4557-b081-2f850d5d1dd0" (UID: "4fbba179-5c67-4557-b081-2f850d5d1dd0"). InnerVolumeSpecName "kube-api-access-fk7zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:50:47.370488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.370453 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbba179-5c67-4557-b081-2f850d5d1dd0-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:47.370488 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.370486 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk7zn\" (UniqueName: \"kubernetes.io/projected/4fbba179-5c67-4557-b081-2f850d5d1dd0-kube-api-access-fk7zn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:50:47.422756 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.422681 5688 generic.go:356] "Generic (PLEG): container finished" podID="4fbba179-5c67-4557-b081-2f850d5d1dd0" containerID="00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991" exitCode=0 Oct 31 18:50:47.422756 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.422739 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" Oct 31 18:50:47.422959 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.422768 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" event={"ID":"4fbba179-5c67-4557-b081-2f850d5d1dd0","Type":"ContainerDied","Data":"00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991"} Oct 31 18:50:47.422959 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.422855 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj" event={"ID":"4fbba179-5c67-4557-b081-2f850d5d1dd0","Type":"ContainerDied","Data":"f79417ea56d4bbabd8984fbbe151746dae30b94db2f2b69bb756e680f252a727"} Oct 31 18:50:47.422959 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.422884 5688 scope.go:122] "RemoveContainer" containerID="00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991" Oct 31 18:50:47.433125 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.433110 5688 scope.go:122] "RemoveContainer" containerID="00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991" Oct 31 18:50:47.433489 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:50:47.433472 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991\": container with ID starting with 00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991 not found: ID does not exist" containerID="00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991" Oct 31 18:50:47.433533 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.433495 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991"} err="failed to get container status \"00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991\": rpc error: code = NotFound desc = could not find container \"00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991\": container with ID starting with 00f0be8f6d898bcbece5eab359e21482261ecf4067e137b94bc67392199ec991 not found: ID does not exist" Oct 31 18:50:47.588376 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.588351 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj"] Oct 31 18:50:47.592948 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:47.592925 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-7b57b7f499-p57nj"] Oct 31 18:50:49.105916 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.105878 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv"] Oct 31 18:50:49.106430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.106416 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4fbba179-5c67-4557-b081-2f850d5d1dd0" containerName="payload-pre-processing" Oct 31 18:50:49.106473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.106432 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbba179-5c67-4557-b081-2f850d5d1dd0" containerName="payload-pre-processing" Oct 31 18:50:49.106614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.106591 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbba179-5c67-4557-b081-2f850d5d1dd0" containerName="payload-pre-processing" Oct 31 18:50:49.111473 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.111451 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.187822 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.187791 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv"] Oct 31 18:50:49.198575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198535 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.198767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198585 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.198767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198617 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.198767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198660 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.198767 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198692 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.198972 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198817 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.198972 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198924 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.199061 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.198983 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.199061 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.199017 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gcc\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.216437 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.216405 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-676224-openshift-default-dockercfg-bfwjg\"" Oct 31 18:50:49.305899 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.305854 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.306173 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.306154 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gcc\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.306341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.306325 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.306440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.306427 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.306545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.306530 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.308618 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.308596 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.315267 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.315420 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.315531 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.315856 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.308513 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.316099 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.308999 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.325236 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.316372 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.328122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.328094 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.330507 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.330485 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.384262 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.384098 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gcc\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.398218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.397239 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token\") pod \"e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.411242 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.408671 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:50:49.428650 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.428622 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:49.568644 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.568613 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbba179-5c67-4557-b081-2f850d5d1dd0" path="/var/lib/kubelet/pods/4fbba179-5c67-4557-b081-2f850d5d1dd0/volumes" Oct 31 18:50:49.871489 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.871466 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv"] Oct 31 18:50:49.874638 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:50:49.874602 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf67da8_9280_46ed_8f9a_0354df5fc2e7.slice/crio-02f74b0e5025a7664f57bf4bb34af4788bc65b0a136f2efec2a0ec2a8ddebcdf WatchSource:0}: Error finding container 02f74b0e5025a7664f57bf4bb34af4788bc65b0a136f2efec2a0ec2a8ddebcdf: Status 404 returned error can't find the container with id 02f74b0e5025a7664f57bf4bb34af4788bc65b0a136f2efec2a0ec2a8ddebcdf Oct 31 18:50:49.877438 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.877399 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:50:49.877505 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.877492 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:50:49.877553 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:49.877538 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:50:50.444909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:50.444862 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" event={"ID":"bbf67da8-9280-46ed-8f9a-0354df5fc2e7","Type":"ContainerStarted","Data":"f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0"} Oct 31 18:50:50.445309 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:50.444949 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" event={"ID":"bbf67da8-9280-46ed-8f9a-0354df5fc2e7","Type":"ContainerStarted","Data":"02f74b0e5025a7664f57bf4bb34af4788bc65b0a136f2efec2a0ec2a8ddebcdf"} Oct 31 18:50:50.561702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:50.561655 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podStartSLOduration=1.5616421379999998 podStartE2EDuration="1.561642138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:50.559417593 +0000 UTC m=+1771.161558735" watchObservedRunningTime="2026-10-31 18:50:50.561642138 +0000 UTC m=+1771.163783282" Oct 31 18:50:51.429696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:51.429665 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:51.434933 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:51.434908 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:51.450689 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:51.450657 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:51.451614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:50:51.451591 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:50:54.904593 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:50:54.904557 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:51:01.043846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:01.043809 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl"] Oct 31 18:51:01.092903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:01.092877 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e11e61-smdcl"] Oct 31 18:51:01.566497 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:01.566471 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417b4306-1468-44ae-b183-66c9d9a8c330" path="/var/lib/kubelet/pods/417b4306-1468-44ae-b183-66c9d9a8c330/volumes" Oct 31 18:51:19.815733 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:19.815700 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:51:19.818311 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:19.816336 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:51:20.484724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:20.484690 5688 scope.go:122] "RemoveContainer" containerID="c0287e0e43d6f90bdc699fd5c702114617ab02b4a6287cf2f44514eda2e02175" Oct 31 18:51:24.905907 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:51:24.905867 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:51:40.307367 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:40.307327 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl"] Oct 31 18:51:40.307946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:40.307569 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" containerID="cri-o://8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0" gracePeriod=30 Oct 31 18:51:43.970244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:43.970213 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8"] Oct 31 18:51:43.974004 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:43.973987 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:43.994881 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:43.994855 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:43.994995 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:43.994915 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:44.077177 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.077149 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a4923\"" Oct 31 18:51:44.077352 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.077335 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a4923-dockercfg-rn6hj\"" Oct 31 18:51:44.079743 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.079719 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8"] Oct 31 18:51:44.096524 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.096499 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:44.096645 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.096538 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:44.097250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.097232 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:44.153220 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.153178 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm\") pod \"payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:44.287164 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.287140 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:44.465481 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.465448 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:51:44.465648 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.465496 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:51:44.659664 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.659640 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8"] Oct 31 18:51:44.661540 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:51:44.661511 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8b4d56_1c09_4ade_8bb6_a0983227c3ab.slice/crio-5490db02e910f0138e8333b07cef70bb7fa5e8eba35d6296e8d29c67bc561936 WatchSource:0}: Error finding container 5490db02e910f0138e8333b07cef70bb7fa5e8eba35d6296e8d29c67bc561936: Status 404 returned error can't find the container with id 5490db02e910f0138e8333b07cef70bb7fa5e8eba35d6296e8d29c67bc561936 Oct 31 18:51:44.663461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.663443 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 18:51:44.795284 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:44.795244 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" event={"ID":"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab","Type":"ContainerStarted","Data":"5490db02e910f0138e8333b07cef70bb7fa5e8eba35d6296e8d29c67bc561936"} Oct 31 18:51:45.606394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.606370 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:51:45.715383 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715354 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715541 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715413 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715541 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715447 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715541 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715508 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715575 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715608 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715769 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715677 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715769 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715738 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.715862 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.715789 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket\") pod \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\" (UID: \"5201c7c1-2539-4fdd-a994-bcf5163ea76d\") " Oct 31 18:51:45.716214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.716158 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:51:45.716575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.716551 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:51:45.716684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.716589 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:51:45.716684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.716615 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:51:45.716684 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.716628 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:51:45.718155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.718122 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:51:45.718353 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.718332 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:51:45.718654 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.718631 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "kube-api-access-ddwhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:51:45.718755 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.718727 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy" pod "5201c7c1-2539-4fdd-a994-bcf5163ea76d" (UID: "5201c7c1-2539-4fdd-a994-bcf5163ea76d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:51:45.806741 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.806260 5688 generic.go:356] "Generic (PLEG): container finished" podID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerID="8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0" exitCode=0 Oct 31 18:51:45.806741 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.806442 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" event={"ID":"5201c7c1-2539-4fdd-a994-bcf5163ea76d","Type":"ContainerDied","Data":"8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0"} Oct 31 18:51:45.807118 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.807096 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" Oct 31 18:51:45.807118 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.807110 5688 scope.go:122] "RemoveContainer" containerID="8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0" Oct 31 18:51:45.807337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.807090 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl" event={"ID":"5201c7c1-2539-4fdd-a994-bcf5163ea76d","Type":"ContainerDied","Data":"28fb48cc74eb75cb202cf7dbbcc5fd276c9159d7629f98aa1cc8352f56258772"} Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817434 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817459 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817474 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817487 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817502 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817517 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5201c7c1-2539-4fdd-a994-bcf5163ea76d-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817530 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817543 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwhj\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-kube-api-access-ddwhj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.817578 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.817557 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5201c7c1-2539-4fdd-a994-bcf5163ea76d-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:51:45.821716 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.821686 5688 scope.go:122] "RemoveContainer" containerID="8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0" Oct 31 18:51:45.822144 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:51:45.822115 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0\": container with ID starting with 8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0 not found: ID does not exist" containerID="8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0" Oct 31 18:51:45.822268 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.822151 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0"} err="failed to get container status \"8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0\": rpc error: code = NotFound desc = could not find container \"8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0\": container with ID starting with 8becc39de3429975962101a52103704bea901ceb218c2855067a5048dad545b0 not found: ID does not exist" Oct 31 18:51:45.933862 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.933775 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl"] Oct 31 18:51:45.936124 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:45.936096 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-wj6dl"] Oct 31 18:51:47.566485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:47.566454 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" path="/var/lib/kubelet/pods/5201c7c1-2539-4fdd-a994-bcf5163ea76d/volumes" Oct 31 18:51:47.821056 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:47.820968 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" event={"ID":"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab","Type":"ContainerStarted","Data":"cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26"} Oct 31 18:51:47.821228 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:47.821085 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:47.937702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:47.937654 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" podStartSLOduration=2.8238834710000003 podStartE2EDuration="4.937641001s" totalImagesPullingTime="2.11375753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:47.934819286 +0000 UTC m=+1828.536960430" watchObservedRunningTime="2026-10-31 18:51:47.937641001 +0000 UTC m=+1828.539782145" Oct 31 18:51:49.827914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.827880 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh"] Oct 31 18:51:49.828398 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.828384 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" Oct 31 18:51:49.828459 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.828401 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" Oct 31 18:51:49.828545 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.828535 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5201c7c1-2539-4fdd-a994-bcf5163ea76d" containerName="istio-proxy" Oct 31 18:51:49.832086 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.832062 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.861507 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.861481 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.861611 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.861514 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsz7t\" (UniqueName: \"kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.861678 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.861631 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.861719 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.861683 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.887110 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.887090 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ceef3b-serving-cert\"" Oct 31 18:51:49.889649 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.889627 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh"] Oct 31 18:51:49.975909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.975875 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.977292 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.977272 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsz7t\" (UniqueName: \"kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.977737 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.977717 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.977962 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.977945 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.980148 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:51:49.977901 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-serving-cert: secret "maas-api-e2e-ipp-b-ceef3b-serving-cert" not found Oct 31 18:51:49.980364 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:51:49.980351 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls podName:bf19e5fb-f26c-421a-8481-d45aac1369ca nodeName:}" failed. No retries permitted until 2026-10-31 18:51:50.480330835 +0000 UTC m=+1831.082471964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls") pod "maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" (UID: "bf19e5fb-f26c-421a-8481-d45aac1369ca") : secret "maas-api-e2e-ipp-b-ceef3b-serving-cert" not found Oct 31 18:51:49.980841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.979812 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:49.995320 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:49.995300 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:50.034559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:50.034537 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsz7t\" (UniqueName: \"kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:50.488793 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:50.488750 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:50.491318 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:50.491289 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:50.748024 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:50.747943 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:51.120921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:51.120895 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh"] Oct 31 18:51:51.123532 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:51:51.123502 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf19e5fb_f26c_421a_8481_d45aac1369ca.slice/crio-9d5c0e7ca70a2e1f87c3f8e95a7ca74cfd88dbb268e17d450c2a44952e8b19fc WatchSource:0}: Error finding container 9d5c0e7ca70a2e1f87c3f8e95a7ca74cfd88dbb268e17d450c2a44952e8b19fc: Status 404 returned error can't find the container with id 9d5c0e7ca70a2e1f87c3f8e95a7ca74cfd88dbb268e17d450c2a44952e8b19fc Oct 31 18:51:51.882052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:51.882009 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" event={"ID":"bf19e5fb-f26c-421a-8481-d45aac1369ca","Type":"ContainerStarted","Data":"9d5c0e7ca70a2e1f87c3f8e95a7ca74cfd88dbb268e17d450c2a44952e8b19fc"} Oct 31 18:51:52.889937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:52.889905 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" event={"ID":"bf19e5fb-f26c-421a-8481-d45aac1369ca","Type":"ContainerStarted","Data":"3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717"} Oct 31 18:51:52.890433 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:52.889986 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:51:53.006898 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:53.006847 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" podStartSLOduration=3.171489702 podStartE2EDuration="4.0068323s" totalImagesPullingTime="835.342598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:53.004715209 +0000 UTC m=+1833.606856352" watchObservedRunningTime="2026-10-31 18:51:53.0068323 +0000 UTC m=+1833.608973443" Oct 31 18:51:54.946062 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:51:54.946027 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:51:58.829047 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:58.829015 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:51:58.901345 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:51:58.901317 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:52:05.685486 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.685453 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24"] Oct 31 18:52:05.689676 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.689655 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.741856 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.741836 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-676224-serving-cert\"" Oct 31 18:52:05.793432 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.793409 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24"] Oct 31 18:52:05.867555 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.867529 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.867697 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.867605 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qgr\" (UniqueName: \"kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.867738 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.867686 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.867788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.867774 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.969320 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.969222 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.969657 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.969636 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qgr\" (UniqueName: \"kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.969833 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.969813 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.970009 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.969992 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.970297 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:52:05.970262 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-serving-cert: secret "maas-api-e2e-ipp-a-676224-serving-cert" not found Oct 31 18:52:05.970414 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:52:05.970350 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls podName:9c330ecb-c9ce-40dc-8ef5-28f708a0ad64 nodeName:}" failed. No retries permitted until 2026-10-31 18:52:06.470326401 +0000 UTC m=+1847.072467523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls") pod "maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" (UID: "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64") : secret "maas-api-e2e-ipp-a-676224-serving-cert" not found Oct 31 18:52:05.970595 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.970572 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:05.972323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:05.972299 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:06.027901 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:06.027872 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qgr\" (UniqueName: \"kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:06.474830 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:06.474799 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:06.477259 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:06.477232 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-676224-6c8947d976-f5p24\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:06.603939 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:06.603910 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:06.976492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:06.976467 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24"] Oct 31 18:52:06.979026 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:52:06.978988 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c330ecb_c9ce_40dc_8ef5_28f708a0ad64.slice/crio-2575b6de56c25d5caea3add31a36ea1d638230488fc50650bb6925a9a02f3d32 WatchSource:0}: Error finding container 2575b6de56c25d5caea3add31a36ea1d638230488fc50650bb6925a9a02f3d32: Status 404 returned error can't find the container with id 2575b6de56c25d5caea3add31a36ea1d638230488fc50650bb6925a9a02f3d32 Oct 31 18:52:06.995008 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:06.994982 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" event={"ID":"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64","Type":"ContainerStarted","Data":"2575b6de56c25d5caea3add31a36ea1d638230488fc50650bb6925a9a02f3d32"} Oct 31 18:52:08.002918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:08.002881 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" event={"ID":"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64","Type":"ContainerStarted","Data":"8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597"} Oct 31 18:52:08.003340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:08.002976 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:08.117462 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:08.117373 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" podStartSLOduration=2.287878081 podStartE2EDuration="3.117358824s" totalImagesPullingTime="829.480743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:52:08.115971613 +0000 UTC m=+1848.718112756" watchObservedRunningTime="2026-10-31 18:52:08.117358824 +0000 UTC m=+1848.719499968" Oct 31 18:52:14.014276 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:14.014248 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 18:52:20.520028 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:20.519995 5688 scope.go:122] "RemoveContainer" containerID="061396d823c191a983d25ac4163298a7ee82ec2c82f4fdd70a069a16028127e8" Oct 31 18:52:24.050148 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:24.050106 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27"] Oct 31 18:52:24.099824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:24.099793 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddd3ef-v8n27"] Oct 31 18:52:24.904881 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:52:24.904840 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:52:25.117250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.115961 5688 generic.go:356] "Generic (PLEG): container finished" podID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerID="b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a" exitCode=0 Oct 31 18:52:25.117250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.116221 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" event={"ID":"294c06f0-8bb4-483a-ba49-b99d8182f7db","Type":"ContainerDied","Data":"b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a"} Oct 31 18:52:25.117250 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.116815 5688 scope.go:122] "RemoveContainer" containerID="b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a" Oct 31 18:52:25.119491 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.119441 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:25.119841 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.119808 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:25.120048 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.120022 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:25.567299 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:25.567263 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248a9706-8640-4ad6-84fd-183182afaf67" path="/var/lib/kubelet/pods/248a9706-8640-4ad6-84fd-183182afaf67/volumes" Oct 31 18:52:26.123394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.123357 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" event={"ID":"294c06f0-8bb4-483a-ba49-b99d8182f7db","Type":"ContainerStarted","Data":"720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d"} Oct 31 18:52:26.700443 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.700400 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht"] Oct 31 18:52:26.706143 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.706119 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.763499 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.763470 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e9c518-openshift-default-dockercfg-fmdtn\"" Oct 31 18:52:26.810760 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.810726 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht"] Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.864770 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.864831 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.864873 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.864931 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.865020 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.865131 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.865170 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.865258 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxk7\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.865290 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.865727 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.901577 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 18:52:26.902909 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.901658 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 18:52:26.969402 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.969300 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.970207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.970160 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.970863 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.970830 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxk7\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.971425 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.970085 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.971863 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.971844 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.972123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.972102 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.972695 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.972678 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.973200 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.973164 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.974097 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.974077 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.975119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.973949 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.975119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.973091 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.975119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.972602 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.975119 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.975071 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.975465 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.974742 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.987828 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.987789 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:26.988811 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:26.988788 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:27.027985 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.027963 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:27.028389 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.028366 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxk7\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7\") pod \"e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:27.319455 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.319427 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:27.857508 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.857470 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 18:52:27.857783 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.857753 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 18:52:27.859571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.859530 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht"] Oct 31 18:52:27.863704 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:52:27.863685 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2135e915_5c0e_4864_99b2_4be140a38237.slice/crio-eeabd0856d582f57900ecc728bee8d92b997d5a17e8684483e3370036146e332 WatchSource:0}: Error finding container eeabd0856d582f57900ecc728bee8d92b997d5a17e8684483e3370036146e332: Status 404 returned error can't find the container with id eeabd0856d582f57900ecc728bee8d92b997d5a17e8684483e3370036146e332 Oct 31 18:52:27.866352 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.866320 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:27.866454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.866393 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:27.866454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:27.866441 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:28.155271 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.154203 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" event={"ID":"2135e915-5c0e-4864-99b2-4be140a38237","Type":"ContainerStarted","Data":"85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159"} Oct 31 18:52:28.155271 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.154268 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" event={"ID":"2135e915-5c0e-4864-99b2-4be140a38237","Type":"ContainerStarted","Data":"eeabd0856d582f57900ecc728bee8d92b997d5a17e8684483e3370036146e332"} Oct 31 18:52:28.272824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.270469 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podStartSLOduration=2.270451199 podStartE2EDuration="2.270451199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:52:28.270019152 +0000 UTC m=+1868.872160296" watchObservedRunningTime="2026-10-31 18:52:28.270451199 +0000 UTC m=+1868.872592344" Oct 31 18:52:28.320260 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.320221 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:28.321788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.321764 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 18:52:28.321922 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.321815 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 18:52:28.854205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.853139 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 18:52:28.854205 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:28.853235 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 18:52:29.324779 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:29.324751 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:29.852489 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:29.852453 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 18:52:29.852864 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:29.852833 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 18:52:30.167969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:30.167879 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:30.169021 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:30.169002 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:52:30.411807 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:30.411769 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:52:30.857454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:30.857411 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 18:52:30.857786 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:30.857730 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 18:52:31.950695 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:31.950453 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:52:31.953906 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:31.953859 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:52:42.238240 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.238178 5688 generic.go:356] "Generic (PLEG): container finished" podID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerID="f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0" exitCode=0 Oct 31 18:52:42.238661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.238250 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" event={"ID":"bbf67da8-9280-46ed-8f9a-0354df5fc2e7","Type":"ContainerDied","Data":"f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0"} Oct 31 18:52:42.238661 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.238604 5688 scope.go:122] "RemoveContainer" containerID="f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0" Oct 31 18:52:42.240017 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.239983 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:42.240114 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.240068 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:42.240177 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.240118 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:52:42.251154 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.251136 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr"] Oct 31 18:52:42.356945 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:42.356902 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt"] Oct 31 18:52:43.245471 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:43.245438 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" event={"ID":"bbf67da8-9280-46ed-8f9a-0354df5fc2e7","Type":"ContainerStarted","Data":"dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a"} Oct 31 18:52:43.452218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:43.449580 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:52:43.452218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:43.449697 5688 patch_prober.go:29] interesting pod/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 18:52:43.452218 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:43.449735 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 18:52:44.429000 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:44.428967 5688 patch_prober.go:29] interesting pod/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 18:52:44.429453 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:44.429023 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 18:52:45.429485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:45.429453 5688 patch_prober.go:29] interesting pod/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 18:52:45.429870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:45.429509 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 18:52:46.429516 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:46.429485 5688 patch_prober.go:29] interesting pod/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 18:52:46.429894 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:46.429540 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 18:52:47.253216 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.252267 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" podUID="fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" containerName="payload-processing" containerID="cri-o://a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7" gracePeriod=25 Oct 31 18:52:47.360433 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.360383 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" podUID="4c9a79be-60c5-4073-a085-a198000c6234" containerName="payload-pre-processing" containerID="cri-o://ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098" gracePeriod=25 Oct 31 18:52:47.429768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.429738 5688 patch_prober.go:29] interesting pod/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 18:52:47.430163 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.429793 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 18:52:47.597549 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.597387 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:52:47.626135 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.626100 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume\") pod \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " Oct 31 18:52:47.626319 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.626213 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx\" (UniqueName: \"kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx\") pod \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\" (UID: \"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3\") " Oct 31 18:52:47.626536 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.626514 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume" pod "fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" (UID: "fb9ca062-3f78-44b2-b90f-fd265dc5b2f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:52:47.626638 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.626624 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:52:47.628603 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.628578 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx" pod "fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" (UID: "fb9ca062-3f78-44b2-b90f-fd265dc5b2f3"). InnerVolumeSpecName "kube-api-access-vblbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:52:47.673761 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.673697 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:52:47.727728 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.727704 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume\") pod \"4c9a79be-60c5-4073-a085-a198000c6234\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " Oct 31 18:52:47.727880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.727815 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g\" (UniqueName: \"kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g\") pod \"4c9a79be-60c5-4073-a085-a198000c6234\" (UID: \"4c9a79be-60c5-4073-a085-a198000c6234\") " Oct 31 18:52:47.728129 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.728105 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume" pod "4c9a79be-60c5-4073-a085-a198000c6234" (UID: "4c9a79be-60c5-4073-a085-a198000c6234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:52:47.728295 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.728279 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblbx\" (UniqueName: \"kubernetes.io/projected/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3-kube-api-access-vblbx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:52:47.728356 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.728300 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a79be-60c5-4073-a085-a198000c6234-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:52:47.730432 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.730409 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g" pod "4c9a79be-60c5-4073-a085-a198000c6234" (UID: "4c9a79be-60c5-4073-a085-a198000c6234"). InnerVolumeSpecName "kube-api-access-5sg7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:52:47.829614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:47.829536 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sg7g\" (UniqueName: \"kubernetes.io/projected/4c9a79be-60c5-4073-a085-a198000c6234-kube-api-access-5sg7g\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:52:48.285996 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.285965 5688 generic.go:356] "Generic (PLEG): container finished" podID="fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" containerID="a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7" exitCode=0 Oct 31 18:52:48.286128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.286016 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" Oct 31 18:52:48.286128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.286045 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" event={"ID":"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3","Type":"ContainerDied","Data":"a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7"} Oct 31 18:52:48.286128 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.286110 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr" event={"ID":"fb9ca062-3f78-44b2-b90f-fd265dc5b2f3","Type":"ContainerDied","Data":"7fc8b62f6bb23efda5323eacb52b7cea2d4b8d31126d4a766e8ddd3f61634f47"} Oct 31 18:52:48.286301 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.286143 5688 scope.go:122] "RemoveContainer" containerID="a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7" Oct 31 18:52:48.287686 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.287663 5688 generic.go:356] "Generic (PLEG): container finished" podID="4c9a79be-60c5-4073-a085-a198000c6234" containerID="ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098" exitCode=0 Oct 31 18:52:48.287792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.287705 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" event={"ID":"4c9a79be-60c5-4073-a085-a198000c6234","Type":"ContainerDied","Data":"ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098"} Oct 31 18:52:48.287792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.287743 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" event={"ID":"4c9a79be-60c5-4073-a085-a198000c6234","Type":"ContainerDied","Data":"1cd6b7785ae621b382cb24956853aeeefa5856ead0064a184709cf478c82b82b"} Oct 31 18:52:48.287877 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.287772 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt" Oct 31 18:52:48.298800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.298782 5688 scope.go:122] "RemoveContainer" containerID="a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7" Oct 31 18:52:48.299034 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:52:48.299016 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7\": container with ID starting with a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7 not found: ID does not exist" containerID="a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7" Oct 31 18:52:48.299084 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.299042 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7"} err="failed to get container status \"a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7\": rpc error: code = NotFound desc = could not find container \"a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7\": container with ID starting with a75f1f3c4c66b5b8ad2c7221acd7689c59bb9ca62d97b287e6d0edd007de27c7 not found: ID does not exist" Oct 31 18:52:48.299084 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.299057 5688 scope.go:122] "RemoveContainer" containerID="ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098" Oct 31 18:52:48.306904 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.306889 5688 scope.go:122] "RemoveContainer" containerID="ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098" Oct 31 18:52:48.307154 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:52:48.307137 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098\": container with ID starting with ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098 not found: ID does not exist" containerID="ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098" Oct 31 18:52:48.307226 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.307162 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098"} err="failed to get container status \"ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098\": rpc error: code = NotFound desc = could not find container \"ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098\": container with ID starting with ac521a8aca0f90a34498e55b4c3fdc9e4690ccf30724c296efad65540381b098 not found: ID does not exist" Oct 31 18:52:48.429596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.429562 5688 patch_prober.go:29] interesting pod/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 18:52:48.429761 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.429620 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 18:52:48.459832 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.459791 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt"] Oct 31 18:52:48.461984 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.461954 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-100faf-7f4b5d8c8c-bhnmt"] Oct 31 18:52:48.597267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.588153 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr"] Oct 31 18:52:48.599843 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:48.599815 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-100faf-6b8d48946c-gdqkr"] Oct 31 18:52:49.433515 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:49.433488 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:52:49.433937 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:49.433918 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:52:49.434521 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:49.434503 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 18:52:49.566915 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:49.566888 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9a79be-60c5-4073-a085-a198000c6234" path="/var/lib/kubelet/pods/4c9a79be-60c5-4073-a085-a198000c6234/volumes" Oct 31 18:52:49.567340 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:49.567325 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" path="/var/lib/kubelet/pods/fb9ca062-3f78-44b2-b90f-fd265dc5b2f3/volumes" Oct 31 18:52:54.905048 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:52:54.905015 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:52:55.119389 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:52:55.119352 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq"] Oct 31 18:53:00.120011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.119911 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" podUID="fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" containerName="payload-processing" containerID="cri-o://db22beffe1f811af9f1d7df15192fd2dd13466fa76ff82a3e7e1986598ef80fc" gracePeriod=25 Oct 31 18:53:00.388614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.388587 5688 generic.go:356] "Generic (PLEG): container finished" podID="fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" containerID="db22beffe1f811af9f1d7df15192fd2dd13466fa76ff82a3e7e1986598ef80fc" exitCode=0 Oct 31 18:53:00.388770 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.388659 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" event={"ID":"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4","Type":"ContainerDied","Data":"db22beffe1f811af9f1d7df15192fd2dd13466fa76ff82a3e7e1986598ef80fc"} Oct 31 18:53:00.388770 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.388721 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" event={"ID":"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4","Type":"ContainerDied","Data":"abc5fdb84a214e9488e9443971fc517260abe20c7d510f9d3c8e8a6e77e66093"} Oct 31 18:53:00.388770 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.388743 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc5fdb84a214e9488e9443971fc517260abe20c7d510f9d3c8e8a6e77e66093" Oct 31 18:53:00.390388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.390367 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:53:00.517679 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.517654 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6\" (UniqueName: \"kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6\") pod \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " Oct 31 18:53:00.517679 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.517682 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume\") pod \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\" (UID: \"fe3a8c90-1fbc-47af-a7cf-11030d7ffea4\") " Oct 31 18:53:00.518124 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.518100 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume" pod "fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" (UID: "fe3a8c90-1fbc-47af-a7cf-11030d7ffea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:53:00.518303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.518284 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:00.519914 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.519893 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6" pod "fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" (UID: "fe3a8c90-1fbc-47af-a7cf-11030d7ffea4"). InnerVolumeSpecName "kube-api-access-6pgp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:53:00.619696 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:00.619668 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pgp6\" (UniqueName: \"kubernetes.io/projected/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4-kube-api-access-6pgp6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:01.395108 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:01.395081 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq" Oct 31 18:53:01.566333 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:01.566309 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq"] Oct 31 18:53:01.612790 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:01.612766 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-64d577d7df-5pnsq"] Oct 31 18:53:03.466393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:03.466346 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc"] Oct 31 18:53:03.466869 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:03.466540 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" containerID="cri-o://12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5" gracePeriod=30 Oct 31 18:53:03.567400 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:03.567370 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" path="/var/lib/kubelet/pods/fe3a8c90-1fbc-47af-a7cf-11030d7ffea4/volumes" Oct 31 18:53:04.452715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:04.452683 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:53:04.452879 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:04.452731 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:53:08.720824 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.720801 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:53:08.803492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803462 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803492 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803493 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803531 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803588 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803636 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803711 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803742 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.803836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803759 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:53:08.803836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803752 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:53:08.803836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803828 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.804069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803883 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy\") pod \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\" (UID: \"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2\") " Oct 31 18:53:08.804069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803942 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:53:08.804069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.803960 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:53:08.804069 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.804035 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:53:08.804580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.804512 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.804580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.804537 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.804580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.804552 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.804580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.804565 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.804580 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.804580 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.806156 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.806132 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:53:08.806289 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.806168 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "kube-api-access-s62hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:53:08.806645 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.806620 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:53:08.806645 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.806622 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy" pod "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" (UID: "42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:53:08.905876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.905846 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62hc\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-kube-api-access-s62hc\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.905876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.905872 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.905876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.905882 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:08.906170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:08.905890 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:53:09.446089 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.446037 5688 generic.go:356] "Generic (PLEG): container finished" podID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerID="12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5" exitCode=0 Oct 31 18:53:09.446089 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.446088 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" Oct 31 18:53:09.446386 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.446127 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" event={"ID":"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2","Type":"ContainerDied","Data":"12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5"} Oct 31 18:53:09.446386 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.446220 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc" event={"ID":"42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2","Type":"ContainerDied","Data":"079cd84837e81c7215105e187308c6a99f138200f4a08e71dc604216c4de0b7c"} Oct 31 18:53:09.446386 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.446253 5688 scope.go:122] "RemoveContainer" containerID="12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5" Oct 31 18:53:09.459113 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.459084 5688 scope.go:122] "RemoveContainer" containerID="12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5" Oct 31 18:53:09.459507 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:53:09.459480 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5\": container with ID starting with 12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5 not found: ID does not exist" containerID="12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5" Oct 31 18:53:09.459600 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.459521 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5"} err="failed to get container status \"12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5\": rpc error: code = NotFound desc = could not find container \"12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5\": container with ID starting with 12dec060f8ba09a2a62fa59aa08fa2de6f0edd1b3d6d56eecde59b3bd11250d5 not found: ID does not exist" Oct 31 18:53:09.609132 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.609103 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc"] Oct 31 18:53:09.658900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:09.658870 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-pf9dc"] Oct 31 18:53:11.566644 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:53:11.566616 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" path="/var/lib/kubelet/pods/42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2/volumes" Oct 31 18:53:24.904931 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:53:24.904896 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:53:54.904387 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:53:54.904353 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:54:16.209349 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:16.209308 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 18:54:16.209805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:16.209394 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 18:54:18.870788 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:18.870754 5688 generic.go:356] "Generic (PLEG): container finished" podID="2135e915-5c0e-4864-99b2-4be140a38237" containerID="85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159" exitCode=0 Oct 31 18:54:18.871364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:18.870834 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" event={"ID":"2135e915-5c0e-4864-99b2-4be140a38237","Type":"ContainerDied","Data":"85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159"} Oct 31 18:54:18.871364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:18.871347 5688 scope.go:122] "RemoveContainer" containerID="85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159" Oct 31 18:54:18.872731 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:18.872703 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:54:18.872801 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:18.872772 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:54:18.872836 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:18.872804 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:54:19.878701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:19.878667 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" event={"ID":"2135e915-5c0e-4864-99b2-4be140a38237","Type":"ContainerStarted","Data":"8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46"} Oct 31 18:54:20.320674 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:20.320646 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:54:20.320868 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:20.320764 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 18:54:20.320868 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:20.320801 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 18:54:20.600323 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:20.600249 5688 scope.go:122] "RemoveContainer" containerID="153a85dacefb75ec9032ea068cfbf8706859484e3cebe8b329db54c6d0a66783" Oct 31 18:54:21.320559 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:21.320529 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 18:54:21.321037 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:21.320594 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 18:54:22.320124 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:22.320086 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 18:54:22.320327 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:22.320154 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 18:54:23.320011 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:23.319980 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 18:54:23.320422 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:23.320036 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 18:54:24.320360 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:24.320319 5688 patch_prober.go:29] interesting pod/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 18:54:24.320743 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:24.320376 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 18:54:24.928038 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:54:24.927982 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:54:25.324222 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:25.324165 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:54:25.324694 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:25.324631 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:54:25.325440 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:25.325422 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:54:40.939889 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.939856 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn"] Oct 31 18:54:40.940329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940315 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4c9a79be-60c5-4073-a085-a198000c6234" containerName="payload-pre-processing" Oct 31 18:54:40.940374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940330 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9a79be-60c5-4073-a085-a198000c6234" containerName="payload-pre-processing" Oct 31 18:54:40.940374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940352 5688 cpu_manager.go:414] "containerMap: removing container" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" Oct 31 18:54:40.940374 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940357 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" Oct 31 18:54:40.940468 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940385 5688 cpu_manager.go:414] "containerMap: removing container" podUID="fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" containerName="payload-processing" Oct 31 18:54:40.940468 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940389 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" containerName="payload-processing" Oct 31 18:54:40.940468 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940398 5688 cpu_manager.go:414] "containerMap: removing container" podUID="fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" containerName="payload-processing" Oct 31 18:54:40.940468 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940403 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" containerName="payload-processing" Oct 31 18:54:40.940587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940473 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9a79be-60c5-4073-a085-a198000c6234" containerName="payload-pre-processing" Oct 31 18:54:40.940587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940484 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ffcf50-6bc2-4af4-9aa3-3c77fded0aa2" containerName="istio-proxy" Oct 31 18:54:40.940587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940491 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3a8c90-1fbc-47af-a7cf-11030d7ffea4" containerName="payload-processing" Oct 31 18:54:40.940587 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.940498 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9ca062-3f78-44b2-b90f-fd265dc5b2f3" containerName="payload-processing" Oct 31 18:54:40.943537 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.943521 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:40.995538 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:40.995510 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-676224\"" Oct 31 18:54:41.045470 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.045437 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-676224-dockercfg-rqsqk\"" Oct 31 18:54:41.045805 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.045769 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.045996 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.045869 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg98v\" (UniqueName: \"kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v\") pod \"payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.047897 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.047871 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn"] Oct 31 18:54:41.147332 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.147304 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg98v\" (UniqueName: \"kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v\") pod \"payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.147480 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.147413 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.148063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.148043 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.203688 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.203633 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg98v\" (UniqueName: \"kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v\") pod \"payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.258664 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.258641 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:41.630207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:41.630165 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn"] Oct 31 18:54:41.632573 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:54:41.632543 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03808518_b1bf_4e9e_bdfd_e49dafc93021.slice/crio-83c31eb7e992e8b6dea3599ed4bf3aaa2d72ae938e3f931404571961a446e67c WatchSource:0}: Error finding container 83c31eb7e992e8b6dea3599ed4bf3aaa2d72ae938e3f931404571961a446e67c: Status 404 returned error can't find the container with id 83c31eb7e992e8b6dea3599ed4bf3aaa2d72ae938e3f931404571961a446e67c Oct 31 18:54:42.007483 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:42.007449 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" event={"ID":"03808518-b1bf-4e9e-bdfd-e49dafc93021","Type":"ContainerStarted","Data":"09cb46df9108ad90ba389b2a4d4f9ac487078bb2bba5095bddab888eb3a8afc0"} Oct 31 18:54:42.007876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:42.007500 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" event={"ID":"03808518-b1bf-4e9e-bdfd-e49dafc93021","Type":"ContainerStarted","Data":"83c31eb7e992e8b6dea3599ed4bf3aaa2d72ae938e3f931404571961a446e67c"} Oct 31 18:54:42.007876 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:42.007543 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:42.120657 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:42.120610 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" podStartSLOduration=2.120596339 podStartE2EDuration="2.120596339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:42.11895191 +0000 UTC m=+2002.721093056" watchObservedRunningTime="2026-10-31 18:54:42.120596339 +0000 UTC m=+2002.722737482" Oct 31 18:54:53.030086 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:54:53.030052 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 18:54:54.904716 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:54:54.904681 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:55:13.579702 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:13.579656 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv"] Oct 31 18:55:18.580778 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:18.580742 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" podUID="0f36eb15-61f5-42b4-9ac4-f6630200d37a" containerName="payload-pre-processing" containerID="cri-o://9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105" gracePeriod=25 Oct 31 18:55:18.844089 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:18.844035 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:55:18.910610 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:18.910583 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume\") pod \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " Oct 31 18:55:18.910763 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:18.910691 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk\" (UniqueName: \"kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk\") pod \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\" (UID: \"0f36eb15-61f5-42b4-9ac4-f6630200d37a\") " Oct 31 18:55:18.910990 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:18.910965 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume" pod "0f36eb15-61f5-42b4-9ac4-f6630200d37a" (UID: "0f36eb15-61f5-42b4-9ac4-f6630200d37a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:55:18.912856 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:18.912834 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk" pod "0f36eb15-61f5-42b4-9ac4-f6630200d37a" (UID: "0f36eb15-61f5-42b4-9ac4-f6630200d37a"). InnerVolumeSpecName "kube-api-access-jk6sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:55:19.011770 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.011743 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk6sk\" (UniqueName: \"kubernetes.io/projected/0f36eb15-61f5-42b4-9ac4-f6630200d37a-kube-api-access-jk6sk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:55:19.011770 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.011769 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f36eb15-61f5-42b4-9ac4-f6630200d37a-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:55:19.255706 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.255669 5688 generic.go:356] "Generic (PLEG): container finished" podID="0f36eb15-61f5-42b4-9ac4-f6630200d37a" containerID="9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105" exitCode=0 Oct 31 18:55:19.255883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.255738 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" Oct 31 18:55:19.255883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.255755 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" event={"ID":"0f36eb15-61f5-42b4-9ac4-f6630200d37a","Type":"ContainerDied","Data":"9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105"} Oct 31 18:55:19.255883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.255838 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv" event={"ID":"0f36eb15-61f5-42b4-9ac4-f6630200d37a","Type":"ContainerDied","Data":"94ba7b5b57b1a28336cbda6446a2b9eb5eebd86978fdcc55dc377554eb0f0e09"} Oct 31 18:55:19.255883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.255864 5688 scope.go:122] "RemoveContainer" containerID="9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105" Oct 31 18:55:19.265121 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.265104 5688 scope.go:122] "RemoveContainer" containerID="9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105" Oct 31 18:55:19.265449 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:55:19.265418 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105\": container with ID starting with 9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105 not found: ID does not exist" containerID="9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105" Oct 31 18:55:19.265530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.265458 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105"} err="failed to get container status \"9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105\": rpc error: code = NotFound desc = could not find container \"9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105\": container with ID starting with 9cf03457762af3f272e750fb9bb4cb3f90378f6bec1a22e4dac257bc53551105 not found: ID does not exist" Oct 31 18:55:19.419134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.419102 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv"] Oct 31 18:55:19.469911 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.469884 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa4693-78dd7ff595-nj8cv"] Oct 31 18:55:19.567597 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:19.567537 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f36eb15-61f5-42b4-9ac4-f6630200d37a" path="/var/lib/kubelet/pods/0f36eb15-61f5-42b4-9ac4-f6630200d37a/volumes" Oct 31 18:55:23.396202 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.396140 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8"] Oct 31 18:55:23.396685 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.396465 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" podUID="fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" containerName="payload-pre-processing" containerID="cri-o://cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26" gracePeriod=30 Oct 31 18:55:23.673752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.673728 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:55:23.764341 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.764316 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm\") pod \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " Oct 31 18:55:23.764478 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.764365 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume\") pod \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\" (UID: \"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab\") " Oct 31 18:55:23.764768 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.764744 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume" pod "fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" (UID: "fa8b4d56-1c09-4ade-8bb6-a0983227c3ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:55:23.764887 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.764871 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:55:23.767072 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.766937 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm" pod "fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" (UID: "fa8b4d56-1c09-4ade-8bb6-a0983227c3ab"). InnerVolumeSpecName "kube-api-access-8m6rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:55:23.872211 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:23.868034 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab-kube-api-access-8m6rm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:55:24.302792 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.302753 5688 generic.go:356] "Generic (PLEG): container finished" podID="fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" containerID="cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26" exitCode=0 Oct 31 18:55:24.303078 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.302825 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" event={"ID":"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab","Type":"ContainerDied","Data":"cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26"} Oct 31 18:55:24.303078 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.302873 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" event={"ID":"fa8b4d56-1c09-4ade-8bb6-a0983227c3ab","Type":"ContainerDied","Data":"5490db02e910f0138e8333b07cef70bb7fa5e8eba35d6296e8d29c67bc561936"} Oct 31 18:55:24.303078 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.302902 5688 scope.go:122] "RemoveContainer" containerID="cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26" Oct 31 18:55:24.303078 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.303032 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8" Oct 31 18:55:24.313625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.313608 5688 scope.go:122] "RemoveContainer" containerID="cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26" Oct 31 18:55:24.313929 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:55:24.313900 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26\": container with ID starting with cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26 not found: ID does not exist" containerID="cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26" Oct 31 18:55:24.313999 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.313942 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26"} err="failed to get container status \"cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26\": rpc error: code = NotFound desc = could not find container \"cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26\": container with ID starting with cdfc8309b763a296a6fd22d0c708dc172b924949c7169bf9e9feb2b1cd177b26 not found: ID does not exist" Oct 31 18:55:24.465348 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.465327 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8"] Oct 31 18:55:24.468829 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:24.468804 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a4923-5466954d75-c4jt8"] Oct 31 18:55:24.904007 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:55:24.903971 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:55:25.565470 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:25.565442 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" path="/var/lib/kubelet/pods/fa8b4d56-1c09-4ade-8bb6-a0983227c3ab/volumes" Oct 31 18:55:54.906972 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:55:54.906929 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:55:55.310384 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:55.310355 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx"] Oct 31 18:55:55.310572 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:55.310547 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" containerID="cri-o://4414ed8c2098c939df182a13a93ffedba06e2279a07f8e7951b77c6c0b9ad88a" gracePeriod=30 Oct 31 18:55:59.468724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:59.468637 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:55:59.468724 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:55:59.468703 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:56:00.505820 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.505703 5688 generic.go:356] "Generic (PLEG): container finished" podID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerID="4414ed8c2098c939df182a13a93ffedba06e2279a07f8e7951b77c6c0b9ad88a" exitCode=0 Oct 31 18:56:00.505820 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.505775 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" event={"ID":"5b235b24-51ac-406e-9f42-6d3196faf32a","Type":"ContainerDied","Data":"4414ed8c2098c939df182a13a93ffedba06e2279a07f8e7951b77c6c0b9ad88a"} Oct 31 18:56:00.632441 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.632414 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:56:00.783394 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783354 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.783575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783407 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.783575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783504 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.783575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783537 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.783754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783596 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.783754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783615 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:00.783754 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783736 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:00.783903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.783876 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:00.784043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.784020 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.784221 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.784103 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.784458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.784252 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.784458 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.784291 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy\") pod \"5b235b24-51ac-406e-9f42-6d3196faf32a\" (UID: \"5b235b24-51ac-406e-9f42-6d3196faf32a\") " Oct 31 18:56:00.784848 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.784818 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:00.785101 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.785076 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:56:00.785175 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.785146 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.785288 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.785165 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.785288 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.785210 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.785288 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.785231 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.786166 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.786133 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:00.786439 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.786416 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:56:00.787122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.787090 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:00.787432 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.787415 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c" pod "5b235b24-51ac-406e-9f42-6d3196faf32a" (UID: "5b235b24-51ac-406e-9f42-6d3196faf32a"). InnerVolumeSpecName "kube-api-access-2qr9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:00.886705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.886621 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.886705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.886665 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.886705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.886681 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b235b24-51ac-406e-9f42-6d3196faf32a-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.886705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.886696 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qr9c\" (UniqueName: \"kubernetes.io/projected/5b235b24-51ac-406e-9f42-6d3196faf32a-kube-api-access-2qr9c\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:00.886953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:00.886710 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b235b24-51ac-406e-9f42-6d3196faf32a-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:01.512918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:01.512886 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" Oct 31 18:56:01.512918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:01.512907 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx" event={"ID":"5b235b24-51ac-406e-9f42-6d3196faf32a","Type":"ContainerDied","Data":"5660cdcc328c3a4c5c54d9ea4a3b8b52a034aa660ea355fe50f58e30d946cd85"} Oct 31 18:56:01.513470 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:01.512967 5688 scope.go:122] "RemoveContainer" containerID="4414ed8c2098c939df182a13a93ffedba06e2279a07f8e7951b77c6c0b9ad88a" Oct 31 18:56:01.727883 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:01.727850 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx"] Oct 31 18:56:01.777385 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:01.777304 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-xrprx"] Oct 31 18:56:03.566080 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:03.566047 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" path="/var/lib/kubelet/pods/5b235b24-51ac-406e-9f42-6d3196faf32a/volumes" Oct 31 18:56:10.108839 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:10.108808 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8"] Oct 31 18:56:10.109260 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:10.109000 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" containerID="cri-o://245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0" gracePeriod=30 Oct 31 18:56:13.224043 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:13.223999 5688 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:56:13.224461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:13.224048 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:56:13.270224 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:13.270195 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt"] Oct 31 18:56:13.270453 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:13.270431 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" containerID="cri-o://da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258" gracePeriod=30 Oct 31 18:56:14.276442 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:14.276394 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-962gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 18:56:14.276977 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:14.276460 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 18:56:15.362530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.362501 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:56:15.475701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475636 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.475701 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475680 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.475874 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475706 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.475874 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475756 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.475874 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475786 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.475874 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475832 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.476063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475907 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.476063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475941 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:15.476063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475970 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:15.476063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.475975 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.476063 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476051 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy\") pod \"c98a2456-b714-49ad-9d8c-acffc22ab681\" (UID: \"c98a2456-b714-49ad-9d8c-acffc22ab681\") " Oct 31 18:56:15.476343 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476053 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:15.476343 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476103 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:15.476343 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476258 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:56:15.476530 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476508 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.476660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476533 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.476660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476551 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.476660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476563 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98a2456-b714-49ad-9d8c-acffc22ab681-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.476660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.476574 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.478123 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.478098 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:56:15.478337 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.478315 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:15.478518 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.478498 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:15.478595 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.478528 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk" pod "c98a2456-b714-49ad-9d8c-acffc22ab681" (UID: "c98a2456-b714-49ad-9d8c-acffc22ab681"). InnerVolumeSpecName "kube-api-access-8d4nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:15.578131 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.578107 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.578269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.578132 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.578269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.578148 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98a2456-b714-49ad-9d8c-acffc22ab681-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.578269 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.578157 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4nk\" (UniqueName: \"kubernetes.io/projected/c98a2456-b714-49ad-9d8c-acffc22ab681-kube-api-access-8d4nk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:15.639364 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.639326 5688 generic.go:356] "Generic (PLEG): container finished" podID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerID="245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0" exitCode=0 Oct 31 18:56:15.639616 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.639593 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" event={"ID":"c98a2456-b714-49ad-9d8c-acffc22ab681","Type":"ContainerDied","Data":"245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0"} Oct 31 18:56:15.639729 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.639686 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" event={"ID":"c98a2456-b714-49ad-9d8c-acffc22ab681","Type":"ContainerDied","Data":"760d4142fc136b80103eaae5d47704180a16a8663b9c00cbccb55e33bf2e26bb"} Oct 31 18:56:15.639729 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.639721 5688 scope.go:122] "RemoveContainer" containerID="245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0" Oct 31 18:56:15.639921 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.639893 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8" Oct 31 18:56:15.649816 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.649798 5688 scope.go:122] "RemoveContainer" containerID="245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0" Oct 31 18:56:15.650078 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:15.650059 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0\": container with ID starting with 245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0 not found: ID does not exist" containerID="245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0" Oct 31 18:56:15.650122 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.650088 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0"} err="failed to get container status \"245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0\": rpc error: code = NotFound desc = could not find container \"245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0\": container with ID starting with 245726dad5d322693a76ffa27cd4ec92b4fd9b31d7c71d7ecad2911e0e21c4e0 not found: ID does not exist" Oct 31 18:56:15.806496 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.806466 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8"] Oct 31 18:56:15.809477 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:15.809448 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-jnpk8"] Oct 31 18:56:17.565845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:17.565814 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" path="/var/lib/kubelet/pods/c98a2456-b714-49ad-9d8c-acffc22ab681/volumes" Oct 31 18:56:18.539309 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.539252 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:56:18.610425 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610399 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610447 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610526 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610564 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610589 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610624 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610666 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610724 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610758 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610770 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:18.611239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610815 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data\") pod \"95eb628e-7cde-44e6-9b9d-21326edc75c9\" (UID: \"95eb628e-7cde-44e6-9b9d-21326edc75c9\") " Oct 31 18:56:18.611239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.610953 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:56:18.611239 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.611141 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:18.611522 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.611503 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.611601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.611529 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.611601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.611545 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.611601 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.611559 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95eb628e-7cde-44e6-9b9d-21326edc75c9-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.611748 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.611729 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:18.612981 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.612945 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:56:18.612981 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.612974 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "kube-api-access-4lzsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:18.613388 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.613368 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:18.613465 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.613385 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token" pod "95eb628e-7cde-44e6-9b9d-21326edc75c9" (UID: "95eb628e-7cde-44e6-9b9d-21326edc75c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:18.658454 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.658432 5688 generic.go:356] "Generic (PLEG): container finished" podID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerID="da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258" exitCode=0 Oct 31 18:56:18.658560 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.658475 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" Oct 31 18:56:18.658560 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.658520 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" event={"ID":"95eb628e-7cde-44e6-9b9d-21326edc75c9","Type":"ContainerDied","Data":"da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258"} Oct 31 18:56:18.658640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.658568 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt" event={"ID":"95eb628e-7cde-44e6-9b9d-21326edc75c9","Type":"ContainerDied","Data":"648ca467ef50dcc7a5a5d77c894624800e45cd0aa0e22a6eca0ea91828ce931a"} Oct 31 18:56:18.658640 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.658589 5688 scope.go:122] "RemoveContainer" containerID="da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258" Oct 31 18:56:18.668344 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.668326 5688 scope.go:122] "RemoveContainer" containerID="da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258" Oct 31 18:56:18.668624 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:18.668606 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258\": container with ID starting with da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258 not found: ID does not exist" containerID="da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258" Oct 31 18:56:18.668667 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.668633 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258"} err="failed to get container status \"da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258\": rpc error: code = NotFound desc = could not find container \"da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258\": container with ID starting with da653542e11debd49aaa6abb60b83f43dd0f46408a5b0f6271d636f359621258 not found: ID does not exist" Oct 31 18:56:18.712267 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.712242 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.712373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.712268 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzsk\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-kube-api-access-4lzsk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.712373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.712283 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.712373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.712291 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95eb628e-7cde-44e6-9b9d-21326edc75c9-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.712373 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.712299 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95eb628e-7cde-44e6-9b9d-21326edc75c9-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:18.818854 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.818778 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt"] Oct 31 18:56:18.824531 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:18.824509 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-962gt"] Oct 31 18:56:19.574689 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:19.574648 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" path="/var/lib/kubelet/pods/95eb628e-7cde-44e6-9b9d-21326edc75c9/volumes" Oct 31 18:56:19.855929 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:19.855852 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:56:19.857705 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:19.857684 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 18:56:24.905578 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:24.905542 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:56:26.750459 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:26.750427 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh"] Oct 31 18:56:26.750974 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:26.750614 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" podUID="bf19e5fb-f26c-421a-8481-d45aac1369ca" containerName="maas-api" containerID="cri-o://3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717" gracePeriod=30 Oct 31 18:56:27.076111 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.076084 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:56:27.219860 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.219831 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs\") pod \"bf19e5fb-f26c-421a-8481-d45aac1369ca\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " Oct 31 18:56:27.220050 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.219889 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t\" (UniqueName: \"kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t\") pod \"bf19e5fb-f26c-421a-8481-d45aac1369ca\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " Oct 31 18:56:27.220050 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.219937 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca\") pod \"bf19e5fb-f26c-421a-8481-d45aac1369ca\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " Oct 31 18:56:27.220172 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.220050 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\") pod \"bf19e5fb-f26c-421a-8481-d45aac1369ca\" (UID: \"bf19e5fb-f26c-421a-8481-d45aac1369ca\") " Oct 31 18:56:27.220900 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.220872 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca" pod "bf19e5fb-f26c-421a-8481-d45aac1369ca" (UID: "bf19e5fb-f26c-421a-8481-d45aac1369ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:56:27.222700 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.222664 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs" pod "bf19e5fb-f26c-421a-8481-d45aac1369ca" (UID: "bf19e5fb-f26c-421a-8481-d45aac1369ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:56:27.223083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.223058 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t" pod "bf19e5fb-f26c-421a-8481-d45aac1369ca" (UID: "bf19e5fb-f26c-421a-8481-d45aac1369ca"). InnerVolumeSpecName "kube-api-access-qsz7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:27.226596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.226567 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls" pod "bf19e5fb-f26c-421a-8481-d45aac1369ca" (UID: "bf19e5fb-f26c-421a-8481-d45aac1369ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 18:56:27.321666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.321592 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:27.321666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.321627 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsz7t\" (UniqueName: \"kubernetes.io/projected/bf19e5fb-f26c-421a-8481-d45aac1369ca-kube-api-access-qsz7t\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:27.321666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.321642 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19e5fb-f26c-421a-8481-d45aac1369ca-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:27.321666 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.321658 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf19e5fb-f26c-421a-8481-d45aac1369ca-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:27.713969 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.713935 5688 generic.go:356] "Generic (PLEG): container finished" podID="bf19e5fb-f26c-421a-8481-d45aac1369ca" containerID="3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717" exitCode=0 Oct 31 18:56:27.714145 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.713998 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" Oct 31 18:56:27.714145 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.713998 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" event={"ID":"bf19e5fb-f26c-421a-8481-d45aac1369ca","Type":"ContainerDied","Data":"3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717"} Oct 31 18:56:27.714255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.714139 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh" event={"ID":"bf19e5fb-f26c-421a-8481-d45aac1369ca","Type":"ContainerDied","Data":"9d5c0e7ca70a2e1f87c3f8e95a7ca74cfd88dbb268e17d450c2a44952e8b19fc"} Oct 31 18:56:27.714255 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.714173 5688 scope.go:122] "RemoveContainer" containerID="3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717" Oct 31 18:56:27.723571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.723548 5688 scope.go:122] "RemoveContainer" containerID="3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717" Oct 31 18:56:27.723892 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:27.723873 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717\": container with ID starting with 3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717 not found: ID does not exist" containerID="3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717" Oct 31 18:56:27.723956 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.723901 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717"} err="failed to get container status \"3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717\": rpc error: code = NotFound desc = could not find container \"3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717\": container with ID starting with 3aed66bac762ac3b6cc6be269087360c73a16131f11f6be9f5fa4321e7c45717 not found: ID does not exist" Oct 31 18:56:27.900285 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.894896 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh"] Oct 31 18:56:27.900285 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:27.894956 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ceef3b-67659f5476-ljcmh"] Oct 31 18:56:29.567571 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:29.567536 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf19e5fb-f26c-421a-8481-d45aac1369ca" path="/var/lib/kubelet/pods/bf19e5fb-f26c-421a-8481-d45aac1369ca/volumes" Oct 31 18:56:36.303244 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:36.303210 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl"] Oct 31 18:56:36.303728 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:36.303484 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" containerID="cri-o://720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d" gracePeriod=30 Oct 31 18:56:41.560092 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.560065 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:56:41.611155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611080 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611155 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611118 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611164 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611209 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611236 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611297 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611345 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611402 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611441 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert\") pod \"294c06f0-8bb4-483a-ba49-b99d8182f7db\" (UID: \"294c06f0-8bb4-483a-ba49-b99d8182f7db\") " Oct 31 18:56:41.611636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611523 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:41.611636 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611560 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:41.611819 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611762 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:41.611870 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611843 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:56:41.611966 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611947 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.612027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611970 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.612027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611983 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.612027 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.611996 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c06f0-8bb4-483a-ba49-b99d8182f7db-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.612484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.612450 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:41.613715 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.613697 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:56:41.613957 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.613940 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:41.614005 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.613973 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "kube-api-access-snqcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:56:41.614005 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.613984 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy" pod "294c06f0-8bb4-483a-ba49-b99d8182f7db" (UID: "294c06f0-8bb4-483a-ba49-b99d8182f7db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:56:41.712632 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.712602 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.712632 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.712627 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.712772 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.712643 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snqcb\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-kube-api-access-snqcb\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.712772 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.712657 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c06f0-8bb4-483a-ba49-b99d8182f7db-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.712772 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.712668 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c06f0-8bb4-483a-ba49-b99d8182f7db-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:56:41.829151 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.829118 5688 generic.go:356] "Generic (PLEG): container finished" podID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerID="720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d" exitCode=0 Oct 31 18:56:41.829339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.829166 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" Oct 31 18:56:41.829339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.829169 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" event={"ID":"294c06f0-8bb4-483a-ba49-b99d8182f7db","Type":"ContainerDied","Data":"720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d"} Oct 31 18:56:41.829339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.829301 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl" event={"ID":"294c06f0-8bb4-483a-ba49-b99d8182f7db","Type":"ContainerDied","Data":"41ae0b2671af28fa1294b212b520f501d632359aac5d931c73d6a16dbfbe4f2a"} Oct 31 18:56:41.829339 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.829325 5688 scope.go:122] "RemoveContainer" containerID="720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d" Oct 31 18:56:41.840433 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.840412 5688 scope.go:122] "RemoveContainer" containerID="b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a" Oct 31 18:56:41.848915 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.848900 5688 scope.go:122] "RemoveContainer" containerID="720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d" Oct 31 18:56:41.849159 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:41.849138 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d\": container with ID starting with 720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d not found: ID does not exist" containerID="720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d" Oct 31 18:56:41.849229 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.849166 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d"} err="failed to get container status \"720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d\": rpc error: code = NotFound desc = could not find container \"720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d\": container with ID starting with 720886d7ab8e8035c27dd4561231e33bce42211a7819e6545c6dce7477a7098d not found: ID does not exist" Oct 31 18:56:41.849229 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.849207 5688 scope.go:122] "RemoveContainer" containerID="b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a" Oct 31 18:56:41.849450 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:41.849435 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a\": container with ID starting with b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a not found: ID does not exist" containerID="b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a" Oct 31 18:56:41.849490 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.849454 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a"} err="failed to get container status \"b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a\": rpc error: code = NotFound desc = could not find container \"b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a\": container with ID starting with b749644361a7c2df92486530122431b2b974121e3dc01db329141ab9f3a6005a not found: ID does not exist" Oct 31 18:56:41.995410 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.995378 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl"] Oct 31 18:56:41.998463 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:41.998440 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ceef3b-openshift-default-589db6db9-v8mgl"] Oct 31 18:56:43.566404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:43.566368 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" path="/var/lib/kubelet/pods/294c06f0-8bb4-483a-ba49-b99d8182f7db/volumes" Oct 31 18:56:54.905664 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:56:54.905627 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:56:56.926066 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:56.926039 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht"] Oct 31 18:56:56.926552 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:56:56.926252 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" containerID="cri-o://8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46" gracePeriod=30 Oct 31 18:57:02.185352 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.185329 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:57:02.267461 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267439 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267505 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267548 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267596 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267583 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267618 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267654 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267709 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267757 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267738 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267845 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs\") pod \"2135e915-5c0e-4864-99b2-4be140a38237\" (UID: \"2135e915-5c0e-4864-99b2-4be140a38237\") " Oct 31 18:57:02.267953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267861 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:57:02.268058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267941 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:57:02.268058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.267954 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:57:02.268058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268016 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:57:02.268234 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268148 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:57:02.268368 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268347 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.268430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268375 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.268560 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268488 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2135e915-5c0e-4864-99b2-4be140a38237-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.268638 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268563 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.268638 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.268576 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.269979 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.269959 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:57:02.270094 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.269989 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:57:02.270361 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.270339 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:57:02.270457 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.270385 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7" pod "2135e915-5c0e-4864-99b2-4be140a38237" (UID: "2135e915-5c0e-4864-99b2-4be140a38237"). InnerVolumeSpecName "kube-api-access-hdxk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:57:02.369653 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.369621 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2135e915-5c0e-4864-99b2-4be140a38237-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.369653 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.369651 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.369880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.369664 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxk7\" (UniqueName: \"kubernetes.io/projected/2135e915-5c0e-4864-99b2-4be140a38237-kube-api-access-hdxk7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.369880 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.369676 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2135e915-5c0e-4864-99b2-4be140a38237-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:57:02.952147 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.952116 5688 generic.go:356] "Generic (PLEG): container finished" podID="2135e915-5c0e-4864-99b2-4be140a38237" containerID="8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46" exitCode=0 Oct 31 18:57:02.952370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.952163 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" Oct 31 18:57:02.952370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.952166 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" event={"ID":"2135e915-5c0e-4864-99b2-4be140a38237","Type":"ContainerDied","Data":"8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46"} Oct 31 18:57:02.952370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.952261 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht" event={"ID":"2135e915-5c0e-4864-99b2-4be140a38237","Type":"ContainerDied","Data":"eeabd0856d582f57900ecc728bee8d92b997d5a17e8684483e3370036146e332"} Oct 31 18:57:02.952370 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.952296 5688 scope.go:122] "RemoveContainer" containerID="8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46" Oct 31 18:57:02.962452 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.962438 5688 scope.go:122] "RemoveContainer" containerID="85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159" Oct 31 18:57:02.970681 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.970664 5688 scope.go:122] "RemoveContainer" containerID="8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46" Oct 31 18:57:02.970926 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:57:02.970906 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46\": container with ID starting with 8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46 not found: ID does not exist" containerID="8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46" Oct 31 18:57:02.970973 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.970935 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46"} err="failed to get container status \"8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46\": rpc error: code = NotFound desc = could not find container \"8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46\": container with ID starting with 8cfb4d5796c41ffcb081f5ffcc021e180cad6735fe133a417b7b534394b82b46 not found: ID does not exist" Oct 31 18:57:02.970973 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.970950 5688 scope.go:122] "RemoveContainer" containerID="85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159" Oct 31 18:57:02.971161 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:57:02.971146 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159\": container with ID starting with 85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159 not found: ID does not exist" containerID="85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159" Oct 31 18:57:02.971238 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:02.971165 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159"} err="failed to get container status \"85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159\": rpc error: code = NotFound desc = could not find container \"85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159\": container with ID starting with 85c6b35df2b8557e9d7b02b563e460b0ded12fd5745834d498a08611d55e4159 not found: ID does not exist" Oct 31 18:57:03.166820 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:03.166790 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht"] Oct 31 18:57:03.215783 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:03.215715 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e9c518-openshift-default-78b796fbf7-sdvht"] Oct 31 18:57:03.565971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:03.565944 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2135e915-5c0e-4864-99b2-4be140a38237" path="/var/lib/kubelet/pods/2135e915-5c0e-4864-99b2-4be140a38237/volumes" Oct 31 18:57:20.658032 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:20.658000 5688 scope.go:122] "RemoveContainer" containerID="db22beffe1f811af9f1d7df15192fd2dd13466fa76ff82a3e7e1986598ef80fc" Oct 31 18:57:24.905787 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:57:24.905744 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:57:54.904760 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:57:54.904725 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:57:59.231513 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.231482 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6"] Oct 31 18:57:59.232246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232125 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" Oct 31 18:57:59.232246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232147 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" Oct 31 18:57:59.232246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232173 5688 cpu_manager.go:414] "containerMap: removing container" podUID="fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" containerName="payload-pre-processing" Oct 31 18:57:59.232246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232219 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" containerName="payload-pre-processing" Oct 31 18:57:59.232246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232241 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0f36eb15-61f5-42b4-9ac4-f6630200d37a" containerName="payload-pre-processing" Oct 31 18:57:59.232246 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232249 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f36eb15-61f5-42b4-9ac4-f6630200d37a" containerName="payload-pre-processing" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232261 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232268 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232286 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232293 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232306 5688 cpu_manager.go:414] "containerMap: removing container" podUID="bf19e5fb-f26c-421a-8481-d45aac1369ca" containerName="maas-api" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232313 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf19e5fb-f26c-421a-8481-d45aac1369ca" containerName="maas-api" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232322 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232328 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232377 5688 cpu_manager.go:414] "containerMap: removing container" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232385 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232398 5688 cpu_manager.go:414] "containerMap: removing container" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232404 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232417 5688 cpu_manager.go:414] "containerMap: removing container" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232423 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232556 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf19e5fb-f26c-421a-8481-d45aac1369ca" containerName="maas-api" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232569 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98a2456-b714-49ad-9d8c-acffc22ab681" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232580 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232588 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232598 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="294c06f0-8bb4-483a-ba49-b99d8182f7db" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232606 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="95eb628e-7cde-44e6-9b9d-21326edc75c9" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232618 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b235b24-51ac-406e-9f42-6d3196faf32a" containerName="istio-proxy" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232635 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8b4d56-1c09-4ade-8bb6-a0983227c3ab" containerName="payload-pre-processing" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232648 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f36eb15-61f5-42b4-9ac4-f6630200d37a" containerName="payload-pre-processing" Oct 31 18:57:59.232660 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.232656 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2135e915-5c0e-4864-99b2-4be140a38237" containerName="istio-proxy" Oct 31 18:57:59.236589 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.236568 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.338497 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.338471 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-919b6e\"" Oct 31 18:57:59.338497 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.338495 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-919b6e-dockercfg-9tvft\"" Oct 31 18:57:59.340468 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.340449 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6"] Oct 31 18:57:59.361573 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.361550 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.361679 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.361585 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztn8\" (UniqueName: \"kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8\") pod \"payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.463168 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.463138 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.463325 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.463285 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztn8\" (UniqueName: \"kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8\") pod \"payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.463965 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.463944 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.519749 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.519726 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztn8\" (UniqueName: \"kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8\") pod \"payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.550614 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.550591 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:57:59.924586 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:57:59.924517 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6"] Oct 31 18:57:59.928416 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:57:59.928388 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4d1022_042d_432b_bae4_43ab46a7d6aa.slice/crio-451b74e4789ef6fcdc46326a52dd254a211314d25759e537459c8e894aeed619 WatchSource:0}: Error finding container 451b74e4789ef6fcdc46326a52dd254a211314d25759e537459c8e894aeed619: Status 404 returned error can't find the container with id 451b74e4789ef6fcdc46326a52dd254a211314d25759e537459c8e894aeed619 Oct 31 18:58:00.249299 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:00.249259 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" event={"ID":"0d4d1022-042d-432b-bae4-43ab46a7d6aa","Type":"ContainerStarted","Data":"ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591"} Oct 31 18:58:00.249687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:00.249321 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" event={"ID":"0d4d1022-042d-432b-bae4-43ab46a7d6aa","Type":"ContainerStarted","Data":"451b74e4789ef6fcdc46326a52dd254a211314d25759e537459c8e894aeed619"} Oct 31 18:58:00.249687 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:00.249456 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:58:00.362635 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:00.362578 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" podStartSLOduration=1.362561035 podStartE2EDuration="1.362561035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:58:00.361806468 +0000 UTC m=+2200.963947611" watchObservedRunningTime="2026-10-31 18:58:00.362561035 +0000 UTC m=+2200.964702181" Oct 31 18:58:11.255393 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:11.255366 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 18:58:24.903140 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:58:24.903108 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:58:43.061512 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.061478 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59"] Oct 31 18:58:43.065335 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.065314 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107173 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107141 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107402 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107379 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107498 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107423 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107620 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107526 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107620 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107554 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107620 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107590 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107670 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107709 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.107752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.107742 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt766\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.156693 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.156659 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e43ccf-openshift-default-dockercfg-vdsvj\"" Oct 31 18:58:43.171697 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.171667 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59"] Oct 31 18:58:43.211493 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.211463 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.212233 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.212206 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.212484 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.212464 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.213052 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.213021 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.213954 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.213923 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.214447 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.214430 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.214647 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.214606 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.214818 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.214800 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt766\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.215039 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.215018 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.215638 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.215620 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.217054 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.214370 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.217219 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.212600 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.217352 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.216041 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.217475 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.213837 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.217591 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.216995 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.221625 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.221602 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.285750 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.285702 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.305404 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.303416 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt766\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766\") pod \"e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.379707 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.379616 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:43.840844 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.840809 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59"] Oct 31 18:58:43.895068 ip-10-0-140-47 kubenswrapper[5688]: W1031 18:58:43.895029 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9edaaa_f008_458b_9b08_c8bd2077d6bc.slice/crio-4ec820212238cec9eded49cf52342953989f7220f1f7f7a12cbeca1c049e6a17 WatchSource:0}: Error finding container 4ec820212238cec9eded49cf52342953989f7220f1f7f7a12cbeca1c049e6a17: Status 404 returned error can't find the container with id 4ec820212238cec9eded49cf52342953989f7220f1f7f7a12cbeca1c049e6a17 Oct 31 18:58:43.897748 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.897711 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:58:43.897846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.897784 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:58:43.897846 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:43.897812 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 18:58:44.485637 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:44.484880 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" event={"ID":"1f9edaaa-f008-458b-9b08-c8bd2077d6bc","Type":"ContainerStarted","Data":"ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8"} Oct 31 18:58:44.485637 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:44.484964 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" event={"ID":"1f9edaaa-f008-458b-9b08-c8bd2077d6bc","Type":"ContainerStarted","Data":"4ec820212238cec9eded49cf52342953989f7220f1f7f7a12cbeca1c049e6a17"} Oct 31 18:58:44.600651 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:44.600587 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podStartSLOduration=1.600569618 podStartE2EDuration="1.600569618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:58:44.599307154 +0000 UTC m=+2245.201448297" watchObservedRunningTime="2026-10-31 18:58:44.600569618 +0000 UTC m=+2245.202710763" Oct 31 18:58:45.380544 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:45.380509 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:45.386058 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:45.386029 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:45.490485 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:45.490456 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:45.491303 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:58:45.491283 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 18:58:54.904971 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:58:54.904925 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:59:10.656329 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:10.656249 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v"] Oct 31 18:59:10.656838 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:10.656496 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" containerID="cri-o://ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f" gracePeriod=30 Oct 31 18:59:15.903979 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:15.903954 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.093494 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.093588 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.093646 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.093690 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.093735 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.093836 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.094110 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.094287 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.094346 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.094454 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token\") pod \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\" (UID: \"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631\") " Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.095495 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.095601 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.095644 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:59:16.097213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.095443 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:16.098083 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.097957 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:16.098278 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098254 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:59:16.098575 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098555 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.098677 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098665 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.098755 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098744 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.098834 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098824 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.098918 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098906 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.098993 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.098981 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.099067 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.099057 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.104430 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.104398 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "kube-api-access-b9rzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:59:16.105642 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.105610 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo" pod "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" (UID: "9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:59:16.200096 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.200067 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9rzj\" (UniqueName: \"kubernetes.io/projected/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-kube-api-access-b9rzj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.200096 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.200093 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:16.655012 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.654979 5688 generic.go:356] "Generic (PLEG): container finished" podID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerID="ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f" exitCode=0 Oct 31 18:59:16.655170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.655045 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" Oct 31 18:59:16.655170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.655066 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" event={"ID":"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631","Type":"ContainerDied","Data":"ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f"} Oct 31 18:59:16.655170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.655118 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v" event={"ID":"9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631","Type":"ContainerDied","Data":"af5a0426b14999e2f31bfc47dc2a40c400bd798272c265971cfba9aca3cd7f7b"} Oct 31 18:59:16.655170 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.655146 5688 scope.go:122] "RemoveContainer" containerID="ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f" Oct 31 18:59:16.664520 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.664502 5688 scope.go:122] "RemoveContainer" containerID="ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f" Oct 31 18:59:16.664779 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:59:16.664756 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f\": container with ID starting with ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f not found: ID does not exist" containerID="ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f" Oct 31 18:59:16.664825 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.664791 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f"} err="failed to get container status \"ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f\": rpc error: code = NotFound desc = could not find container \"ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f\": container with ID starting with ca16868d546450383e27762ca850561234e2faecbf88cfbf8268bb7eed7ba14f not found: ID does not exist" Oct 31 18:59:16.817903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.817875 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v"] Oct 31 18:59:16.867525 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:16.867497 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-mck4v"] Oct 31 18:59:17.565752 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:17.565724 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" path="/var/lib/kubelet/pods/9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631/volumes" Oct 31 18:59:21.323834 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:21.323801 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287"] Oct 31 18:59:21.324334 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:21.323991 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" containerID="cri-o://6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4" gracePeriod=30 Oct 31 18:59:24.918116 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:59:24.918077 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 18:59:26.580562 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.580536 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:59:26.602798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.602736 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.602798 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.602784 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.602971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.602875 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.602971 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.602913 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.603074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.602999 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.603074 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603030 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.603214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603075 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.603214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603119 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.603214 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603158 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token\") pod \"63e77c87-aa2d-40d8-b273-c1547f9b6479\" (UID: \"63e77c87-aa2d-40d8-b273-c1547f9b6479\") " Oct 31 18:59:26.603369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603115 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:26.603369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603292 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:26.603369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603143 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:59:26.603369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603277 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:26.603369 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603362 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:26.603627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603576 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.603627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603595 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.603627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603609 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.603627 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603622 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e77c87-aa2d-40d8-b273-c1547f9b6479-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.603763 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.603636 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.605734 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.605707 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 18:59:26.605845 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.605746 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 18:59:26.605903 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.605853 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "kube-api-access-j9ms6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:59:26.605946 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.605918 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token" pod "63e77c87-aa2d-40d8-b273-c1547f9b6479" (UID: "63e77c87-aa2d-40d8-b273-c1547f9b6479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:59:26.704975 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.704956 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.704975 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.704976 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.705134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.704986 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ms6\" (UniqueName: \"kubernetes.io/projected/63e77c87-aa2d-40d8-b273-c1547f9b6479-kube-api-access-j9ms6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.705134 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.704996 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e77c87-aa2d-40d8-b273-c1547f9b6479-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 18:59:26.710142 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.710117 5688 generic.go:356] "Generic (PLEG): container finished" podID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerID="6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4" exitCode=0 Oct 31 18:59:26.710260 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.710166 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" Oct 31 18:59:26.710260 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.710208 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" event={"ID":"63e77c87-aa2d-40d8-b273-c1547f9b6479","Type":"ContainerDied","Data":"6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4"} Oct 31 18:59:26.710381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.710272 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287" event={"ID":"63e77c87-aa2d-40d8-b273-c1547f9b6479","Type":"ContainerDied","Data":"a618dcd382344d7e2c23ad235ffe3ed944138c1ff9fdb57a1469aaeaae4a9683"} Oct 31 18:59:26.710381 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.710305 5688 scope.go:122] "RemoveContainer" containerID="6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4" Oct 31 18:59:26.719542 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.719528 5688 scope.go:122] "RemoveContainer" containerID="6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4" Oct 31 18:59:26.719798 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:59:26.719782 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4\": container with ID starting with 6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4 not found: ID does not exist" containerID="6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4" Oct 31 18:59:26.719861 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.719809 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4"} err="failed to get container status \"6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4\": rpc error: code = NotFound desc = could not find container \"6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4\": container with ID starting with 6a6173b2537ae459b19b3c73a573f8b6c4f263add360395379db233181d7eca4 not found: ID does not exist" Oct 31 18:59:26.858213 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.858111 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287"] Oct 31 18:59:26.861875 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:26.861851 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-jh287"] Oct 31 18:59:27.565953 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:27.565926 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" path="/var/lib/kubelet/pods/63e77c87-aa2d-40d8-b273-c1547f9b6479/volumes" Oct 31 18:59:46.096207 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:46.096156 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6_0d4d1022-042d-432b-bae4-43ab46a7d6aa/payload-processing/0.log" Oct 31 18:59:46.312044 ip-10-0-140-47 kubenswrapper[5688]: I1031 18:59:46.312020 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6_0d4d1022-042d-432b-bae4-43ab46a7d6aa/payload-processing/0.log" Oct 31 18:59:54.904032 ip-10-0-140-47 kubenswrapper[5688]: E1031 18:59:54.903997 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:00:00.160529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.160489 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6"] Oct 31 19:00:00.161155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.161030 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" Oct 31 19:00:00.161155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.161049 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" Oct 31 19:00:00.161155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.161085 5688 cpu_manager.go:414] "containerMap: removing container" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" Oct 31 19:00:00.161155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.161092 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" Oct 31 19:00:00.161437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.161248 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e77c87-aa2d-40d8-b273-c1547f9b6479" containerName="istio-proxy" Oct 31 19:00:00.161437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.161265 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe524cd-93c5-4ad0-b7a6-f5c8f21c0631" containerName="istio-proxy" Oct 31 19:00:00.164427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.164407 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj"] Oct 31 19:00:00.164561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.164540 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.167597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.167577 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89"] Oct 31 19:00:00.167712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.167675 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.171431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.171411 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.209928 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.209908 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw"] Oct 31 19:00:00.212078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.212057 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4b2\" (UniqueName: \"kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2\") pod \"maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.212235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.212090 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.213341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.213325 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z"] Oct 31 19:00:00.213442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.213428 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.216421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.216407 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:00:00.216707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.216685 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc"] Oct 31 19:00:00.216800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.216784 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.220059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.220043 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.266221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.264635 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6"] Oct 31 19:00:00.266924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.266903 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89"] Oct 31 19:00:00.268490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.268469 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj"] Oct 31 19:00:00.313111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.313056 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4b2\" (UniqueName: \"kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2\") pod \"maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.316673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.316644 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.317582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.317353 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mfv\" (UniqueName: \"kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv\") pod \"maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.317582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.317436 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf22r\" (UniqueName: \"kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r\") pod \"maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.318000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.317970 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891220-xlt89\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.318101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318037 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.318101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318077 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.318255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318111 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfjt\" (UniqueName: \"kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt\") pod \"maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.318255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318141 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.318255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318168 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89m5b\" (UniqueName: \"kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.318255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318243 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88sx\" (UniqueName: \"kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx\") pod \"maas-api-key-cleanup-29891220-xlt89\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.318547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318284 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z"] Oct 31 19:00:00.318547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318341 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc"] Oct 31 19:00:00.318547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318291 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.319008 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.318981 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.366255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.366231 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw"] Oct 31 19:00:00.371096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.371071 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4b2\" (UniqueName: \"kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2\") pod \"maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.419353 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419292 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf22r\" (UniqueName: \"kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r\") pod \"maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.419353 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419338 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891220-xlt89\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419363 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419384 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419403 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfjt\" (UniqueName: \"kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt\") pod \"maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419418 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419433 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89m5b\" (UniqueName: \"kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419472 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88sx\" (UniqueName: \"kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx\") pod \"maas-api-key-cleanup-29891220-xlt89\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.419534 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419512 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.419890 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.419618 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mfv\" (UniqueName: \"kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv\") pod \"maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.420138 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.420104 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891220-xlt89\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.420275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.420257 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.420275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.420267 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.420368 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.420257 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.420368 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.420333 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.476420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.476390 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89m5b\" (UniqueName: \"kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.477104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.477073 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88sx\" (UniqueName: \"kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx\") pod \"maas-api-key-cleanup-29891220-xlt89\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.477224 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.477153 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfjt\" (UniqueName: \"kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt\") pod \"maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.477342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.477327 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf22r\" (UniqueName: \"kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r\") pod \"maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.477515 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.477494 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mfv\" (UniqueName: \"kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv\") pod \"maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.480144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.480130 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:00.486119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.486100 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:00.491692 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.491662 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:00.529072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.528154 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:00.533694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.533466 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:00.543710 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.543687 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:00.868254 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.868231 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6"] Oct 31 19:00:00.870983 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:00.870956 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44d7d5d_926e_46ca_836f_76b5dc7fa009.slice/crio-32875ab98e37ff8da02e11bde97cacdbc30916f99f977cb058b9edfa86bc2c4c WatchSource:0}: Error finding container 32875ab98e37ff8da02e11bde97cacdbc30916f99f977cb058b9edfa86bc2c4c: Status 404 returned error can't find the container with id 32875ab98e37ff8da02e11bde97cacdbc30916f99f977cb058b9edfa86bc2c4c Oct 31 19:00:00.888769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.888744 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerStarted","Data":"32875ab98e37ff8da02e11bde97cacdbc30916f99f977cb058b9edfa86bc2c4c"} Oct 31 19:00:00.914146 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.914126 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89"] Oct 31 19:00:00.915787 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:00.915762 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e56fc5e_eca7_44a8_9824_0e1fbc8ac1b9.slice/crio-16ccb9ec38de9a9732fd376644f4a708558246985dc22b58b7357e0deab791f4 WatchSource:0}: Error finding container 16ccb9ec38de9a9732fd376644f4a708558246985dc22b58b7357e0deab791f4: Status 404 returned error can't find the container with id 16ccb9ec38de9a9732fd376644f4a708558246985dc22b58b7357e0deab791f4 Oct 31 19:00:00.967159 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.967074 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw"] Oct 31 19:00:00.970978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.970953 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc"] Oct 31 19:00:00.971240 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:00.971214 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebc3615_4215_4b83_8909_93d05364df57.slice/crio-a98e653f00daab5fc1ca41c6bf0257338c8e909c34e422438d698fc6a4280fd6 WatchSource:0}: Error finding container a98e653f00daab5fc1ca41c6bf0257338c8e909c34e422438d698fc6a4280fd6: Status 404 returned error can't find the container with id a98e653f00daab5fc1ca41c6bf0257338c8e909c34e422438d698fc6a4280fd6 Oct 31 19:00:00.973748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:00.973727 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z"] Oct 31 19:00:00.975201 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:00.975086 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cd622b_ad09_459d_8c9e_7f0c76e5be24.slice/crio-b62d7b85393f3b9c85bf8d197170a932798990535aff87ed095b006f174ceaba WatchSource:0}: Error finding container b62d7b85393f3b9c85bf8d197170a932798990535aff87ed095b006f174ceaba: Status 404 returned error can't find the container with id b62d7b85393f3b9c85bf8d197170a932798990535aff87ed095b006f174ceaba Oct 31 19:00:00.977828 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:00.977696 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdda03cf_d2d9_4a8b_8dc4_c49b84474913.slice/crio-1990c4e736d1d11d5c4decb1a53901f7173001fe47fea4068bb3bef1a8f5f743 WatchSource:0}: Error finding container 1990c4e736d1d11d5c4decb1a53901f7173001fe47fea4068bb3bef1a8f5f743: Status 404 returned error can't find the container with id 1990c4e736d1d11d5c4decb1a53901f7173001fe47fea4068bb3bef1a8f5f743 Oct 31 19:00:01.032826 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.032794 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj"] Oct 31 19:00:01.036012 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:01.035976 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4ee0d5_d505_428e_a6b7_479af7c9f35f.slice/crio-8f51306fc0bd2e9384266e822e2cc5f1597d313a43ad905d28e38950e1487dae WatchSource:0}: Error finding container 8f51306fc0bd2e9384266e822e2cc5f1597d313a43ad905d28e38950e1487dae: Status 404 returned error can't find the container with id 8f51306fc0bd2e9384266e822e2cc5f1597d313a43ad905d28e38950e1487dae Oct 31 19:00:01.895855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.895818 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerStarted","Data":"dbc81b795570184072b5d97d7594a24e40e7be231ec9900b53c3e666042d22d0"} Oct 31 19:00:01.896234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.895881 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerStarted","Data":"b62d7b85393f3b9c85bf8d197170a932798990535aff87ed095b006f174ceaba"} Oct 31 19:00:01.897460 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.897436 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerStarted","Data":"5a2a2f8bb8b91757205a6858eaa74443c601344d1e85e8dbcfd6081a033a8b8b"} Oct 31 19:00:01.897588 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.897480 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerStarted","Data":"a98e653f00daab5fc1ca41c6bf0257338c8e909c34e422438d698fc6a4280fd6"} Oct 31 19:00:01.898955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.898931 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerStarted","Data":"87fa71d721670d6e2681d380272204fec1ddf215599e27b36fc120d232705a02"} Oct 31 19:00:01.899044 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.898977 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerStarted","Data":"8f51306fc0bd2e9384266e822e2cc5f1597d313a43ad905d28e38950e1487dae"} Oct 31 19:00:01.900425 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.900404 5688 generic.go:356] "Generic (PLEG): container finished" podID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerID="c61eb6b310fa703dcdf79de4cae2b44c67190f36678d3bcbc11a5144530b4e6a" exitCode=60 Oct 31 19:00:01.900508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.900468 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerDied","Data":"c61eb6b310fa703dcdf79de4cae2b44c67190f36678d3bcbc11a5144530b4e6a"} Oct 31 19:00:01.900624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.900609 5688 scope.go:122] "RemoveContainer" containerID="c61eb6b310fa703dcdf79de4cae2b44c67190f36678d3bcbc11a5144530b4e6a" Oct 31 19:00:01.902031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.902014 5688 generic.go:356] "Generic (PLEG): container finished" podID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerID="eac09ff6e4e716414847c59894b567c674038575392a9ec821da3f6b8972a09d" exitCode=60 Oct 31 19:00:01.902117 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.902097 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" event={"ID":"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9","Type":"ContainerDied","Data":"eac09ff6e4e716414847c59894b567c674038575392a9ec821da3f6b8972a09d"} Oct 31 19:00:01.902172 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.902158 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" event={"ID":"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9","Type":"ContainerStarted","Data":"16ccb9ec38de9a9732fd376644f4a708558246985dc22b58b7357e0deab791f4"} Oct 31 19:00:01.902282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.902266 5688 scope.go:122] "RemoveContainer" containerID="eac09ff6e4e716414847c59894b567c674038575392a9ec821da3f6b8972a09d" Oct 31 19:00:01.903651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.903602 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" event={"ID":"cdda03cf-d2d9-4a8b-8dc4-c49b84474913","Type":"ContainerStarted","Data":"e3a8bd019d6eb3dfb739cceb41e2dcd2d625a783ae0d9224a5f157e3169282fd"} Oct 31 19:00:01.903651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:01.903637 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" event={"ID":"cdda03cf-d2d9-4a8b-8dc4-c49b84474913","Type":"ContainerStarted","Data":"1990c4e736d1d11d5c4decb1a53901f7173001fe47fea4068bb3bef1a8f5f743"} Oct 31 19:00:02.007294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.007242 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" podStartSLOduration=2.007222944 podStartE2EDuration="2.007222944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.006485479 +0000 UTC m=+2322.608626621" watchObservedRunningTime="2026-10-31 19:00:02.007222944 +0000 UTC m=+2322.609364088" Oct 31 19:00:02.229875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.229831 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" podStartSLOduration=2.229818153 podStartE2EDuration="2.229818153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.22850895 +0000 UTC m=+2322.830650096" watchObservedRunningTime="2026-10-31 19:00:02.229818153 +0000 UTC m=+2322.831959344" Oct 31 19:00:02.452494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.452400 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" podStartSLOduration=2.452381038 podStartE2EDuration="2.452381038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.450835456 +0000 UTC m=+2323.052976599" watchObservedRunningTime="2026-10-31 19:00:02.452381038 +0000 UTC m=+2323.054522182" Oct 31 19:00:02.582248 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.582174 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" podStartSLOduration=2.582159758 podStartE2EDuration="2.582159758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.579569307 +0000 UTC m=+2323.181710454" watchObservedRunningTime="2026-10-31 19:00:02.582159758 +0000 UTC m=+2323.184300919" Oct 31 19:00:02.911086 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.911055 5688 generic.go:356] "Generic (PLEG): container finished" podID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerID="dbc81b795570184072b5d97d7594a24e40e7be231ec9900b53c3e666042d22d0" exitCode=60 Oct 31 19:00:02.911506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.911119 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerDied","Data":"dbc81b795570184072b5d97d7594a24e40e7be231ec9900b53c3e666042d22d0"} Oct 31 19:00:02.911506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.911376 5688 scope.go:122] "RemoveContainer" containerID="dbc81b795570184072b5d97d7594a24e40e7be231ec9900b53c3e666042d22d0" Oct 31 19:00:02.912772 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.912739 5688 generic.go:356] "Generic (PLEG): container finished" podID="4ebc3615-4215-4b83-8909-93d05364df57" containerID="5a2a2f8bb8b91757205a6858eaa74443c601344d1e85e8dbcfd6081a033a8b8b" exitCode=60 Oct 31 19:00:02.912855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.912834 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerDied","Data":"5a2a2f8bb8b91757205a6858eaa74443c601344d1e85e8dbcfd6081a033a8b8b"} Oct 31 19:00:02.913024 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.913004 5688 scope.go:122] "RemoveContainer" containerID="5a2a2f8bb8b91757205a6858eaa74443c601344d1e85e8dbcfd6081a033a8b8b" Oct 31 19:00:02.914584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.914561 5688 generic.go:356] "Generic (PLEG): container finished" podID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerID="87fa71d721670d6e2681d380272204fec1ddf215599e27b36fc120d232705a02" exitCode=60 Oct 31 19:00:02.914662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.914645 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerDied","Data":"87fa71d721670d6e2681d380272204fec1ddf215599e27b36fc120d232705a02"} Oct 31 19:00:02.914865 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.914848 5688 scope.go:122] "RemoveContainer" containerID="87fa71d721670d6e2681d380272204fec1ddf215599e27b36fc120d232705a02" Oct 31 19:00:02.916600 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.916583 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerStarted","Data":"6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af"} Oct 31 19:00:02.918540 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.918496 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" event={"ID":"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9","Type":"ContainerStarted","Data":"82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f"} Oct 31 19:00:02.920015 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.919992 5688 generic.go:356] "Generic (PLEG): container finished" podID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerID="e3a8bd019d6eb3dfb739cceb41e2dcd2d625a783ae0d9224a5f157e3169282fd" exitCode=60 Oct 31 19:00:02.920139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.920056 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" event={"ID":"cdda03cf-d2d9-4a8b-8dc4-c49b84474913","Type":"ContainerDied","Data":"e3a8bd019d6eb3dfb739cceb41e2dcd2d625a783ae0d9224a5f157e3169282fd"} Oct 31 19:00:02.920283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:02.920268 5688 scope.go:122] "RemoveContainer" containerID="e3a8bd019d6eb3dfb739cceb41e2dcd2d625a783ae0d9224a5f157e3169282fd" Oct 31 19:00:03.359010 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.358970 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" podStartSLOduration=3.3589572260000002 podStartE2EDuration="3.358957226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:03.356553625 +0000 UTC m=+2323.958694768" watchObservedRunningTime="2026-10-31 19:00:03.358957226 +0000 UTC m=+2323.961098366" Oct 31 19:00:03.580639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.580585 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" podStartSLOduration=3.5805719099999997 podStartE2EDuration="3.58057191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:03.579505609 +0000 UTC m=+2324.181646754" watchObservedRunningTime="2026-10-31 19:00:03.58057191 +0000 UTC m=+2324.182713054" Oct 31 19:00:03.937712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.937681 5688 generic.go:356] "Generic (PLEG): container finished" podID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerID="6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af" exitCode=60 Oct 31 19:00:03.938092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.937755 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerDied","Data":"6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af"} Oct 31 19:00:03.938092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.937874 5688 scope.go:122] "RemoveContainer" containerID="c61eb6b310fa703dcdf79de4cae2b44c67190f36678d3bcbc11a5144530b4e6a" Oct 31 19:00:03.938092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.937991 5688 scope.go:122] "RemoveContainer" containerID="6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af" Oct 31 19:00:03.938326 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:03.938277 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6_odh-ai-gateway-infra(e44d7d5d-926e-46ca-836f-76b5dc7fa009)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" Oct 31 19:00:03.940385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.940095 5688 generic.go:356] "Generic (PLEG): container finished" podID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerID="82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f" exitCode=60 Oct 31 19:00:03.940385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.940239 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" event={"ID":"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9","Type":"ContainerDied","Data":"82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f"} Oct 31 19:00:03.940600 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.940583 5688 scope.go:122] "RemoveContainer" containerID="82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f" Oct 31 19:00:03.940861 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:03.940839 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891220-xlt89_odh-ai-gateway-infra(4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" Oct 31 19:00:03.942343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.942319 5688 generic.go:356] "Generic (PLEG): container finished" podID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerID="595ff3c56826924d87b13cf36c5934655d3bbb08f325b53d95abf8ec066c4d0e" exitCode=60 Oct 31 19:00:03.942492 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.942467 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" event={"ID":"cdda03cf-d2d9-4a8b-8dc4-c49b84474913","Type":"ContainerDied","Data":"595ff3c56826924d87b13cf36c5934655d3bbb08f325b53d95abf8ec066c4d0e"} Oct 31 19:00:03.942639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.942621 5688 scope.go:122] "RemoveContainer" containerID="595ff3c56826924d87b13cf36c5934655d3bbb08f325b53d95abf8ec066c4d0e" Oct 31 19:00:03.942874 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:03.942853 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc_odh-ai-gateway-infra(cdda03cf-d2d9-4a8b-8dc4-c49b84474913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" Oct 31 19:00:03.944625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.944604 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerStarted","Data":"98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679"} Oct 31 19:00:03.944823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.944809 5688 scope.go:122] "RemoveContainer" containerID="98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679" Oct 31 19:00:03.945029 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:03.945007 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z_odh-ai-gateway-infra(e4cd622b-ad09-459d-8c9e-7f0c76e5be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" Oct 31 19:00:03.946502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.946481 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerStarted","Data":"a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088"} Oct 31 19:00:03.946659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.946646 5688 scope.go:122] "RemoveContainer" containerID="a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088" Oct 31 19:00:03.946857 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:03.946837 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw_odh-ai-gateway-infra(4ebc3615-4215-4b83-8909-93d05364df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" podUID="4ebc3615-4215-4b83-8909-93d05364df57" Oct 31 19:00:03.948407 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.948387 5688 generic.go:356] "Generic (PLEG): container finished" podID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerID="e5571c9b9501fad509b1427853aa3c4d32bd11f737232e825ad191fb627603a8" exitCode=60 Oct 31 19:00:03.948482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.948424 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerDied","Data":"e5571c9b9501fad509b1427853aa3c4d32bd11f737232e825ad191fb627603a8"} Oct 31 19:00:03.948634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:03.948621 5688 scope.go:122] "RemoveContainer" containerID="e5571c9b9501fad509b1427853aa3c4d32bd11f737232e825ad191fb627603a8" Oct 31 19:00:03.948799 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:03.948785 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj_odh-ai-gateway-infra(0c4ee0d5-d505-428e-a6b7-479af7c9f35f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" Oct 31 19:00:04.091409 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.091385 5688 scope.go:122] "RemoveContainer" containerID="eac09ff6e4e716414847c59894b567c674038575392a9ec821da3f6b8972a09d" Oct 31 19:00:04.106112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.106094 5688 scope.go:122] "RemoveContainer" containerID="e3a8bd019d6eb3dfb739cceb41e2dcd2d625a783ae0d9224a5f157e3169282fd" Oct 31 19:00:04.118475 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.118457 5688 scope.go:122] "RemoveContainer" containerID="87fa71d721670d6e2681d380272204fec1ddf215599e27b36fc120d232705a02" Oct 31 19:00:04.956490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.956457 5688 generic.go:356] "Generic (PLEG): container finished" podID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerID="98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679" exitCode=60 Oct 31 19:00:04.956970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.956547 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerDied","Data":"98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679"} Oct 31 19:00:04.956970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.956612 5688 scope.go:122] "RemoveContainer" containerID="dbc81b795570184072b5d97d7594a24e40e7be231ec9900b53c3e666042d22d0" Oct 31 19:00:04.956970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.956856 5688 scope.go:122] "RemoveContainer" containerID="98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679" Oct 31 19:00:04.957105 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:04.957087 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z_odh-ai-gateway-infra(e4cd622b-ad09-459d-8c9e-7f0c76e5be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" Oct 31 19:00:04.958882 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.958858 5688 generic.go:356] "Generic (PLEG): container finished" podID="4ebc3615-4215-4b83-8909-93d05364df57" containerID="a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088" exitCode=60 Oct 31 19:00:04.958966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.958933 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerDied","Data":"a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088"} Oct 31 19:00:04.959257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.959237 5688 scope.go:122] "RemoveContainer" containerID="a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088" Oct 31 19:00:04.959498 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:04.959478 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw_odh-ai-gateway-infra(4ebc3615-4215-4b83-8909-93d05364df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" podUID="4ebc3615-4215-4b83-8909-93d05364df57" Oct 31 19:00:04.960695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.960677 5688 scope.go:122] "RemoveContainer" containerID="e5571c9b9501fad509b1427853aa3c4d32bd11f737232e825ad191fb627603a8" Oct 31 19:00:04.960917 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:04.960899 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj_odh-ai-gateway-infra(0c4ee0d5-d505-428e-a6b7-479af7c9f35f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" Oct 31 19:00:04.962175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.962156 5688 scope.go:122] "RemoveContainer" containerID="6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af" Oct 31 19:00:04.962397 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:04.962379 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6_odh-ai-gateway-infra(e44d7d5d-926e-46ca-836f-76b5dc7fa009)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" Oct 31 19:00:04.963535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.963519 5688 scope.go:122] "RemoveContainer" containerID="82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f" Oct 31 19:00:04.963748 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:04.963724 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891220-xlt89_odh-ai-gateway-infra(4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" Oct 31 19:00:04.964926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.964911 5688 scope.go:122] "RemoveContainer" containerID="595ff3c56826924d87b13cf36c5934655d3bbb08f325b53d95abf8ec066c4d0e" Oct 31 19:00:04.965065 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:04.965050 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc_odh-ai-gateway-infra(cdda03cf-d2d9-4a8b-8dc4-c49b84474913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" Oct 31 19:00:04.969963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:04.969931 5688 scope.go:122] "RemoveContainer" containerID="5a2a2f8bb8b91757205a6858eaa74443c601344d1e85e8dbcfd6081a033a8b8b" Oct 31 19:00:05.971891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:05.971865 5688 scope.go:122] "RemoveContainer" containerID="98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679" Oct 31 19:00:05.972507 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:05.972156 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z_odh-ai-gateway-infra(e4cd622b-ad09-459d-8c9e-7f0c76e5be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" Oct 31 19:00:05.973833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:05.973814 5688 scope.go:122] "RemoveContainer" containerID="a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088" Oct 31 19:00:05.974072 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:05.974049 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw_odh-ai-gateway-infra(4ebc3615-4215-4b83-8909-93d05364df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" podUID="4ebc3615-4215-4b83-8909-93d05364df57" Oct 31 19:00:14.017775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:14.017709 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerID="ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8" exitCode=0 Oct 31 19:00:14.017775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:14.017758 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" event={"ID":"1f9edaaa-f008-458b-9b08-c8bd2077d6bc","Type":"ContainerDied","Data":"ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8"} Oct 31 19:00:14.018211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:14.018084 5688 scope.go:122] "RemoveContainer" containerID="ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8" Oct 31 19:00:14.019467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:14.019434 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:00:14.019581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:14.019517 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:00:14.019581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:14.019566 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:00:15.025749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.025715 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" event={"ID":"1f9edaaa-f008-458b-9b08-c8bd2077d6bc","Type":"ContainerStarted","Data":"4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a"} Oct 31 19:00:15.379965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.379887 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 19:00:15.381400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.381372 5688 patch_prober.go:29] interesting pod/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 19:00:15.381553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.381428 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 19:00:15.490454 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.490422 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 19:00:15.560647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.560619 5688 scope.go:122] "RemoveContainer" containerID="595ff3c56826924d87b13cf36c5934655d3bbb08f325b53d95abf8ec066c4d0e" Oct 31 19:00:15.561792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.561767 5688 scope.go:122] "RemoveContainer" containerID="82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f" Oct 31 19:00:15.562061 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.562040 5688 scope.go:122] "RemoveContainer" containerID="a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088" Oct 31 19:00:15.562799 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:15.562786 5688 scope.go:122] "RemoveContainer" containerID="98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679" Oct 31 19:00:16.058226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:16.057222 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerStarted","Data":"6a4c0b1ecda4cc414992d4dcec9782e2557efa0d6e08a382de5729d9bff2a414"} Oct 31 19:00:16.380442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:16.380404 5688 patch_prober.go:29] interesting pod/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 19:00:16.380613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:16.380451 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 19:00:17.062604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.062570 5688 generic.go:356] "Generic (PLEG): container finished" podID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerID="ced7a524cd47567c7b984cc507d5aa555f8b120d9958fae02f6f768ce2eca6d4" exitCode=60 Oct 31 19:00:17.063050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.062647 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerDied","Data":"ced7a524cd47567c7b984cc507d5aa555f8b120d9958fae02f6f768ce2eca6d4"} Oct 31 19:00:17.063050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.062706 5688 scope.go:122] "RemoveContainer" containerID="98692a18f604764b0b52e2bddf489cc40e2e435461d397595a0e52fadde11679" Oct 31 19:00:17.063050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.063045 5688 scope.go:122] "RemoveContainer" containerID="ced7a524cd47567c7b984cc507d5aa555f8b120d9958fae02f6f768ce2eca6d4" Oct 31 19:00:17.063366 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:17.063336 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z_odh-ai-gateway-infra(e4cd622b-ad09-459d-8c9e-7f0c76e5be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" Oct 31 19:00:17.064609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.064590 5688 generic.go:356] "Generic (PLEG): container finished" podID="4ebc3615-4215-4b83-8909-93d05364df57" containerID="6a4c0b1ecda4cc414992d4dcec9782e2557efa0d6e08a382de5729d9bff2a414" exitCode=60 Oct 31 19:00:17.064691 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.064616 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerDied","Data":"6a4c0b1ecda4cc414992d4dcec9782e2557efa0d6e08a382de5729d9bff2a414"} Oct 31 19:00:17.064942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.064927 5688 scope.go:122] "RemoveContainer" containerID="6a4c0b1ecda4cc414992d4dcec9782e2557efa0d6e08a382de5729d9bff2a414" Oct 31 19:00:17.065195 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:17.065161 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw_odh-ai-gateway-infra(4ebc3615-4215-4b83-8909-93d05364df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" podUID="4ebc3615-4215-4b83-8909-93d05364df57" Oct 31 19:00:17.066430 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.066413 5688 generic.go:356] "Generic (PLEG): container finished" podID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerID="22fe935e5ac6374aeed7d2ef0a0331857e0dda277faac5eb8549953fac57dac3" exitCode=60 Oct 31 19:00:17.066536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.066490 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" event={"ID":"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9","Type":"ContainerDied","Data":"22fe935e5ac6374aeed7d2ef0a0331857e0dda277faac5eb8549953fac57dac3"} Oct 31 19:00:17.066787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.066773 5688 scope.go:122] "RemoveContainer" containerID="22fe935e5ac6374aeed7d2ef0a0331857e0dda277faac5eb8549953fac57dac3" Oct 31 19:00:17.067007 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:17.066968 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891220-xlt89_odh-ai-gateway-infra(4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" Oct 31 19:00:17.068242 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.068224 5688 generic.go:356] "Generic (PLEG): container finished" podID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerID="1e1244c1eb4e04837f1ea6554080d36ff8ec0fa9733839284a5b066a8ae5c46b" exitCode=60 Oct 31 19:00:17.068310 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.068292 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" event={"ID":"cdda03cf-d2d9-4a8b-8dc4-c49b84474913","Type":"ContainerDied","Data":"1e1244c1eb4e04837f1ea6554080d36ff8ec0fa9733839284a5b066a8ae5c46b"} Oct 31 19:00:17.068675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.068662 5688 scope.go:122] "RemoveContainer" containerID="1e1244c1eb4e04837f1ea6554080d36ff8ec0fa9733839284a5b066a8ae5c46b" Oct 31 19:00:17.068853 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:17.068838 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc_odh-ai-gateway-infra(cdda03cf-d2d9-4a8b-8dc4-c49b84474913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" Oct 31 19:00:17.074784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.074768 5688 scope.go:122] "RemoveContainer" containerID="a1381b40447918e52234e6798821d16442b9d6a87c2bb8ac5184aa2e025bd088" Oct 31 19:00:17.088294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.088274 5688 scope.go:122] "RemoveContainer" containerID="82953901b3f0665aa66a0efa0be0e3d16a029a6ed44a045abb534f11561dea5f" Oct 31 19:00:17.099895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.099878 5688 scope.go:122] "RemoveContainer" containerID="595ff3c56826924d87b13cf36c5934655d3bbb08f325b53d95abf8ec066c4d0e" Oct 31 19:00:17.227596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.227566 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw"] Oct 31 19:00:17.380232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.380104 5688 patch_prober.go:29] interesting pod/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 19:00:17.380232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.380170 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 19:00:17.559404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.559378 5688 scope.go:122] "RemoveContainer" containerID="e5571c9b9501fad509b1427853aa3c4d32bd11f737232e825ad191fb627603a8" Oct 31 19:00:17.559568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:17.559448 5688 scope.go:122] "RemoveContainer" containerID="6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af" Oct 31 19:00:18.082067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.082040 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerStarted","Data":"47d6e85143e60fe311b1ffe22215368e1268a47ed10c0e2cac7b255271d20e19"} Oct 31 19:00:18.084059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.084039 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerStarted","Data":"5be8e60ba6b2b73b172f12af46d7b2df2d18bde683184c10aa7c682e32061f0e"} Oct 31 19:00:18.188482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.188454 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z"] Oct 31 19:00:18.276858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.276836 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:18.299752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.299727 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89"] Oct 31 19:00:18.379909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.379884 5688 patch_prober.go:29] interesting pod/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 19:00:18.379973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.379937 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 19:00:18.400224 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.400172 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:18.414859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.414668 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r\" (UniqueName: \"kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r\") pod \"4ebc3615-4215-4b83-8909-93d05364df57\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " Oct 31 19:00:18.414859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.414849 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca\") pod \"4ebc3615-4215-4b83-8909-93d05364df57\" (UID: \"4ebc3615-4215-4b83-8909-93d05364df57\") " Oct 31 19:00:18.415767 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.415743 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca" pod "4ebc3615-4215-4b83-8909-93d05364df57" (UID: "4ebc3615-4215-4b83-8909-93d05364df57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:18.420520 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.420486 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r" pod "4ebc3615-4215-4b83-8909-93d05364df57" (UID: "4ebc3615-4215-4b83-8909-93d05364df57"). InnerVolumeSpecName "kube-api-access-qf22r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:18.478868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.478845 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:18.516557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.516534 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt\" (UniqueName: \"kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt\") pod \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " Oct 31 19:00:18.516706 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.516679 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca\") pod \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\" (UID: \"e4cd622b-ad09-459d-8c9e-7f0c76e5be24\") " Oct 31 19:00:18.517045 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.517023 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ebc3615-4215-4b83-8909-93d05364df57-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.517124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.517042 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca" pod "e4cd622b-ad09-459d-8c9e-7f0c76e5be24" (UID: "e4cd622b-ad09-459d-8c9e-7f0c76e5be24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:18.517124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.517050 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf22r\" (UniqueName: \"kubernetes.io/projected/4ebc3615-4215-4b83-8909-93d05364df57-kube-api-access-qf22r\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.518842 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.518815 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt" pod "e4cd622b-ad09-459d-8c9e-7f0c76e5be24" (UID: "e4cd622b-ad09-459d-8c9e-7f0c76e5be24"). InnerVolumeSpecName "kube-api-access-ltfjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:18.529628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.529607 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc"] Oct 31 19:00:18.618490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.618397 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx\" (UniqueName: \"kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx\") pod \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " Oct 31 19:00:18.619381 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.619060 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca\") pod \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\" (UID: \"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9\") " Oct 31 19:00:18.619509 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.619464 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca" pod "4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" (UID: "4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:18.619677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.619657 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltfjt\" (UniqueName: \"kubernetes.io/projected/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-kube-api-access-ltfjt\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.619742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.619684 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.619742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.619700 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd622b-ad09-459d-8c9e-7f0c76e5be24-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.621981 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.621955 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx" pod "4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" (UID: "4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9"). InnerVolumeSpecName "kube-api-access-p88sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:18.722999 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.721107 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88sx\" (UniqueName: \"kubernetes.io/projected/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9-kube-api-access-p88sx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.730365 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.730341 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:18.923557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.923433 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca\") pod \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " Oct 31 19:00:18.923557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.923495 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv\" (UniqueName: \"kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv\") pod \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\" (UID: \"cdda03cf-d2d9-4a8b-8dc4-c49b84474913\") " Oct 31 19:00:18.923912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.923887 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca" pod "cdda03cf-d2d9-4a8b-8dc4-c49b84474913" (UID: "cdda03cf-d2d9-4a8b-8dc4-c49b84474913"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:18.924173 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.924037 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:18.926579 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:18.926552 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv" pod "cdda03cf-d2d9-4a8b-8dc4-c49b84474913" (UID: "cdda03cf-d2d9-4a8b-8dc4-c49b84474913"). InnerVolumeSpecName "kube-api-access-b8mfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:19.025098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.025071 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8mfv\" (UniqueName: \"kubernetes.io/projected/cdda03cf-d2d9-4a8b-8dc4-c49b84474913-kube-api-access-b8mfv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:19.095513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.095486 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" Oct 31 19:00:19.095513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.095511 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z" event={"ID":"e4cd622b-ad09-459d-8c9e-7f0c76e5be24","Type":"ContainerDied","Data":"b62d7b85393f3b9c85bf8d197170a932798990535aff87ed095b006f174ceaba"} Oct 31 19:00:19.096000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.095571 5688 scope.go:122] "RemoveContainer" containerID="ced7a524cd47567c7b984cc507d5aa555f8b120d9958fae02f6f768ce2eca6d4" Oct 31 19:00:19.097308 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.097281 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" event={"ID":"4ebc3615-4215-4b83-8909-93d05364df57","Type":"ContainerDied","Data":"a98e653f00daab5fc1ca41c6bf0257338c8e909c34e422438d698fc6a4280fd6"} Oct 31 19:00:19.097422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.097378 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw" Oct 31 19:00:19.111811 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.111791 5688 scope.go:122] "RemoveContainer" containerID="6a4c0b1ecda4cc414992d4dcec9782e2557efa0d6e08a382de5729d9bff2a414" Oct 31 19:00:19.112360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.112338 5688 generic.go:356] "Generic (PLEG): container finished" podID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerID="47d6e85143e60fe311b1ffe22215368e1268a47ed10c0e2cac7b255271d20e19" exitCode=60 Oct 31 19:00:19.112972 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.112419 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerDied","Data":"47d6e85143e60fe311b1ffe22215368e1268a47ed10c0e2cac7b255271d20e19"} Oct 31 19:00:19.113390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.113371 5688 scope.go:122] "RemoveContainer" containerID="47d6e85143e60fe311b1ffe22215368e1268a47ed10c0e2cac7b255271d20e19" Oct 31 19:00:19.113997 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:19.113964 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj_odh-ai-gateway-infra(0c4ee0d5-d505-428e-a6b7-479af7c9f35f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" Oct 31 19:00:19.115485 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.115463 5688 generic.go:356] "Generic (PLEG): container finished" podID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerID="5be8e60ba6b2b73b172f12af46d7b2df2d18bde683184c10aa7c682e32061f0e" exitCode=60 Oct 31 19:00:19.115583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.115529 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerDied","Data":"5be8e60ba6b2b73b172f12af46d7b2df2d18bde683184c10aa7c682e32061f0e"} Oct 31 19:00:19.115950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.115929 5688 scope.go:122] "RemoveContainer" containerID="5be8e60ba6b2b73b172f12af46d7b2df2d18bde683184c10aa7c682e32061f0e" Oct 31 19:00:19.116247 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:19.116216 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6_odh-ai-gateway-infra(e44d7d5d-926e-46ca-836f-76b5dc7fa009)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" Oct 31 19:00:19.116996 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.116975 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" Oct 31 19:00:19.117079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.117002 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89" event={"ID":"4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9","Type":"ContainerDied","Data":"16ccb9ec38de9a9732fd376644f4a708558246985dc22b58b7357e0deab791f4"} Oct 31 19:00:19.118413 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.118396 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" event={"ID":"cdda03cf-d2d9-4a8b-8dc4-c49b84474913","Type":"ContainerDied","Data":"1990c4e736d1d11d5c4decb1a53901f7173001fe47fea4068bb3bef1a8f5f743"} Oct 31 19:00:19.118502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.118410 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc" Oct 31 19:00:19.121808 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.121790 5688 scope.go:122] "RemoveContainer" containerID="e5571c9b9501fad509b1427853aa3c4d32bd11f737232e825ad191fb627603a8" Oct 31 19:00:19.157479 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.157451 5688 scope.go:122] "RemoveContainer" containerID="6a41bec29a3101be4c3dbedd7dc545385fd9b23f7924f6ffc11aefb3d39439af" Oct 31 19:00:19.171074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.171054 5688 scope.go:122] "RemoveContainer" containerID="22fe935e5ac6374aeed7d2ef0a0331857e0dda277faac5eb8549953fac57dac3" Oct 31 19:00:19.182362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.182344 5688 scope.go:122] "RemoveContainer" containerID="1e1244c1eb4e04837f1ea6554080d36ff8ec0fa9733839284a5b066a8ae5c46b" Oct 31 19:00:19.286898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.286866 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z"] Oct 31 19:00:19.295631 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.295607 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-713bf0-29891220-mkw9z"] Oct 31 19:00:19.358209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.358037 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6"] Oct 31 19:00:19.380729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.380701 5688 patch_prober.go:29] interesting pod/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 19:00:19.380854 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.380759 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 19:00:19.416505 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.416473 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89"] Oct 31 19:00:19.418023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.418005 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891220-xlt89"] Oct 31 19:00:19.456467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.456394 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj"] Oct 31 19:00:19.525201 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.525160 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw"] Oct 31 19:00:19.527867 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.527845 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e43ccf-29891220-mxmpw"] Oct 31 19:00:19.566779 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.566748 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" path="/var/lib/kubelet/pods/4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9/volumes" Oct 31 19:00:19.567198 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.567163 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebc3615-4215-4b83-8909-93d05364df57" path="/var/lib/kubelet/pods/4ebc3615-4215-4b83-8909-93d05364df57/volumes" Oct 31 19:00:19.567583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.567564 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" path="/var/lib/kubelet/pods/e4cd622b-ad09-459d-8c9e-7f0c76e5be24/volumes" Oct 31 19:00:19.767531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.767498 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc"] Oct 31 19:00:19.769052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:19.769031 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3932ef-29891220-x9tfc"] Oct 31 19:00:20.295807 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.295787 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:20.297790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.297774 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:20.384628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.384569 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 19:00:20.385416 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.385399 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 19:00:20.437818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.437794 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca\") pod \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " Oct 31 19:00:20.437930 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.437883 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b\" (UniqueName: \"kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b\") pod \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " Oct 31 19:00:20.437991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.437932 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca\") pod \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\" (UID: \"0c4ee0d5-d505-428e-a6b7-479af7c9f35f\") " Oct 31 19:00:20.438042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.437998 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2\" (UniqueName: \"kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2\") pod \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\" (UID: \"e44d7d5d-926e-46ca-836f-76b5dc7fa009\") " Oct 31 19:00:20.438254 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.438221 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca" pod "e44d7d5d-926e-46ca-836f-76b5dc7fa009" (UID: "e44d7d5d-926e-46ca-836f-76b5dc7fa009"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:20.438376 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.438354 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca" pod "0c4ee0d5-d505-428e-a6b7-479af7c9f35f" (UID: "0c4ee0d5-d505-428e-a6b7-479af7c9f35f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:20.440141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.440105 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b" pod "0c4ee0d5-d505-428e-a6b7-479af7c9f35f" (UID: "0c4ee0d5-d505-428e-a6b7-479af7c9f35f"). InnerVolumeSpecName "kube-api-access-89m5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:20.440271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.440166 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2" pod "e44d7d5d-926e-46ca-836f-76b5dc7fa009" (UID: "e44d7d5d-926e-46ca-836f-76b5dc7fa009"). InnerVolumeSpecName "kube-api-access-8x4b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:20.553128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.546461 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:20.553128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.546648 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4b2\" (UniqueName: \"kubernetes.io/projected/e44d7d5d-926e-46ca-836f-76b5dc7fa009-kube-api-access-8x4b2\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:20.553128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.546666 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d7d5d-926e-46ca-836f-76b5dc7fa009-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:20.553128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:20.546680 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89m5b\" (UniqueName: \"kubernetes.io/projected/0c4ee0d5-d505-428e-a6b7-479af7c9f35f-kube-api-access-89m5b\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:21.138116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.138090 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" Oct 31 19:00:21.138300 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.138087 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6" event={"ID":"e44d7d5d-926e-46ca-836f-76b5dc7fa009","Type":"ContainerDied","Data":"32875ab98e37ff8da02e11bde97cacdbc30916f99f977cb058b9edfa86bc2c4c"} Oct 31 19:00:21.138300 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.138282 5688 scope.go:122] "RemoveContainer" containerID="5be8e60ba6b2b73b172f12af46d7b2df2d18bde683184c10aa7c682e32061f0e" Oct 31 19:00:21.139479 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.139446 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" Oct 31 19:00:21.139541 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.139469 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj" event={"ID":"0c4ee0d5-d505-428e-a6b7-479af7c9f35f","Type":"ContainerDied","Data":"8f51306fc0bd2e9384266e822e2cc5f1597d313a43ad905d28e38950e1487dae"} Oct 31 19:00:21.147247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.147231 5688 scope.go:122] "RemoveContainer" containerID="47d6e85143e60fe311b1ffe22215368e1268a47ed10c0e2cac7b255271d20e19" Oct 31 19:00:21.300677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.300647 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj"] Oct 31 19:00:21.306656 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.306633 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891220-vsmmj"] Oct 31 19:00:21.415923 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.415857 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6"] Oct 31 19:00:21.420757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.420730 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-919b6e-29891220-jkvv6"] Oct 31 19:00:21.565535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.565509 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" path="/var/lib/kubelet/pods/0c4ee0d5-d505-428e-a6b7-479af7c9f35f/volumes" Oct 31 19:00:21.565888 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.565875 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" path="/var/lib/kubelet/pods/cdda03cf-d2d9-4a8b-8dc4-c49b84474913/volumes" Oct 31 19:00:21.566222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:21.566209 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" path="/var/lib/kubelet/pods/e44d7d5d-926e-46ca-836f-76b5dc7fa009/volumes" Oct 31 19:00:24.905685 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:24.905653 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:00:26.374213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374166 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz"] Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374593 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374603 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374617 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374622 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374633 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374638 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374646 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374650 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374663 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374667 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374676 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374681 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374686 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374690 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374699 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374703 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374710 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374715 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374720 5688 cpu_manager.go:414] "containerMap: removing container" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374726 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374732 5688 cpu_manager.go:414] "containerMap: removing container" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374736 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374741 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374745 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374751 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374756 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374763 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374767 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374775 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374779 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374787 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374792 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374859 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374866 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.374863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374874 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374879 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374886 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374891 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374896 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374901 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374906 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374911 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374916 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374921 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:26.375649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.374926 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:26.379543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.379527 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.407660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.407638 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.407776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.407755 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b2fq\" (UniqueName: \"kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq\") pod \"payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.482122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.482094 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-713bf0\"" Oct 31 19:00:26.482122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.482097 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-713bf0-dockercfg-lnmd6\"" Oct 31 19:00:26.484016 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.483991 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz"] Oct 31 19:00:26.508515 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.508490 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.508615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.508587 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b2fq\" (UniqueName: \"kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq\") pod \"payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.509137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.509120 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.564930 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.564906 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b2fq\" (UniqueName: \"kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq\") pod \"payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:26.695200 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:26.695122 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:27.077848 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:27.077810 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz"] Oct 31 19:00:27.081386 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:27.081355 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e38397_012b_4b6c_b2a7_8e0681d7c894.slice/crio-2afe28869bf0dfdd65489966bfeed5106311856c1d4835d021256cdfb505b68c WatchSource:0}: Error finding container 2afe28869bf0dfdd65489966bfeed5106311856c1d4835d021256cdfb505b68c: Status 404 returned error can't find the container with id 2afe28869bf0dfdd65489966bfeed5106311856c1d4835d021256cdfb505b68c Oct 31 19:00:27.175817 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:27.175781 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" event={"ID":"81e38397-012b-4b6c-b2a7-8e0681d7c894","Type":"ContainerStarted","Data":"32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8"} Oct 31 19:00:27.175956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:27.175850 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:27.175956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:27.175879 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" event={"ID":"81e38397-012b-4b6c-b2a7-8e0681d7c894","Type":"ContainerStarted","Data":"2afe28869bf0dfdd65489966bfeed5106311856c1d4835d021256cdfb505b68c"} Oct 31 19:00:27.291112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:27.291063 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" podStartSLOduration=1.291049761 podStartE2EDuration="1.291049761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:27.288872282 +0000 UTC m=+2347.891013428" watchObservedRunningTime="2026-10-31 19:00:27.291049761 +0000 UTC m=+2347.893190905" Oct 31 19:00:38.183221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:38.183175 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:00:46.521276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.521243 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24"] Oct 31 19:00:46.521715 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.521420 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" podUID="9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" containerName="maas-api" containerID="cri-o://8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597" gracePeriod=30 Oct 31 19:00:46.827452 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.827420 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 19:00:46.919618 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.919587 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs\") pod \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " Oct 31 19:00:46.919816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.919657 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\") pod \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " Oct 31 19:00:46.919888 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.919828 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr\" (UniqueName: \"kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr\") pod \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " Oct 31 19:00:46.919888 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.919878 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca\") pod \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\" (UID: \"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64\") " Oct 31 19:00:46.920494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.920326 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca" pod "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" (UID: "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:46.921955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.921925 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs" pod "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" (UID: "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:00:46.922065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.921994 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls" pod "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" (UID: "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:00:46.922361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:46.922344 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr" pod "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" (UID: "9c330ecb-c9ce-40dc-8ef5-28f708a0ad64"). InnerVolumeSpecName "kube-api-access-r9qgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:47.021662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.021630 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9qgr\" (UniqueName: \"kubernetes.io/projected/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-kube-api-access-r9qgr\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:47.021662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.021658 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:47.021662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.021668 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:47.021886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.021677 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:47.030895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.030857 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn"] Oct 31 19:00:47.031088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.031054 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" podUID="03808518-b1bf-4e9e-bdfd-e49dafc93021" containerName="payload-pre-processing" containerID="cri-o://09cb46df9108ad90ba389b2a4d4f9ac487078bb2bba5095bddab888eb3a8afc0" gracePeriod=30 Oct 31 19:00:47.277218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.277162 5688 generic.go:356] "Generic (PLEG): container finished" podID="9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" containerID="8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597" exitCode=0 Oct 31 19:00:47.277399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.277291 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" Oct 31 19:00:47.277399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.277313 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" event={"ID":"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64","Type":"ContainerDied","Data":"8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597"} Oct 31 19:00:47.277507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.277408 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24" event={"ID":"9c330ecb-c9ce-40dc-8ef5-28f708a0ad64","Type":"ContainerDied","Data":"2575b6de56c25d5caea3add31a36ea1d638230488fc50650bb6925a9a02f3d32"} Oct 31 19:00:47.277507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.277441 5688 scope.go:122] "RemoveContainer" containerID="8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597" Oct 31 19:00:47.279121 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.279096 5688 generic.go:356] "Generic (PLEG): container finished" podID="03808518-b1bf-4e9e-bdfd-e49dafc93021" containerID="09cb46df9108ad90ba389b2a4d4f9ac487078bb2bba5095bddab888eb3a8afc0" exitCode=0 Oct 31 19:00:47.279280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.279165 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" event={"ID":"03808518-b1bf-4e9e-bdfd-e49dafc93021","Type":"ContainerDied","Data":"09cb46df9108ad90ba389b2a4d4f9ac487078bb2bba5095bddab888eb3a8afc0"} Oct 31 19:00:47.288699 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.288684 5688 scope.go:122] "RemoveContainer" containerID="8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597" Oct 31 19:00:47.288974 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:47.288955 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597\": container with ID starting with 8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597 not found: ID does not exist" containerID="8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597" Oct 31 19:00:47.289045 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.288983 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597"} err="failed to get container status \"8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597\": rpc error: code = NotFound desc = could not find container \"8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597\": container with ID starting with 8c49b06565af1af40ea2228ca1d671b175e75ada7eb55c3b9738916024ff1597 not found: ID does not exist" Oct 31 19:00:47.306045 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.306027 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 19:00:47.324547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.324525 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume\") pod \"03808518-b1bf-4e9e-bdfd-e49dafc93021\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " Oct 31 19:00:47.324709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.324694 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v\" (UniqueName: \"kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v\") pod \"03808518-b1bf-4e9e-bdfd-e49dafc93021\" (UID: \"03808518-b1bf-4e9e-bdfd-e49dafc93021\") " Oct 31 19:00:47.324864 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.324832 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume" pod "03808518-b1bf-4e9e-bdfd-e49dafc93021" (UID: "03808518-b1bf-4e9e-bdfd-e49dafc93021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:00:47.324980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.324944 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03808518-b1bf-4e9e-bdfd-e49dafc93021-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:47.326703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.326683 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v" pod "03808518-b1bf-4e9e-bdfd-e49dafc93021" (UID: "03808518-b1bf-4e9e-bdfd-e49dafc93021"). InnerVolumeSpecName "kube-api-access-sg98v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:00:47.426420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.426358 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg98v\" (UniqueName: \"kubernetes.io/projected/03808518-b1bf-4e9e-bdfd-e49dafc93021-kube-api-access-sg98v\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:00:47.440002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.439975 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24"] Oct 31 19:00:47.444912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.444885 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-676224-6c8947d976-f5p24"] Oct 31 19:00:47.581731 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:47.581690 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" path="/var/lib/kubelet/pods/9c330ecb-c9ce-40dc-8ef5-28f708a0ad64/volumes" Oct 31 19:00:48.286589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.286558 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" event={"ID":"03808518-b1bf-4e9e-bdfd-e49dafc93021","Type":"ContainerDied","Data":"83c31eb7e992e8b6dea3599ed4bf3aaa2d72ae938e3f931404571961a446e67c"} Oct 31 19:00:48.286589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.286587 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn" Oct 31 19:00:48.286817 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.286609 5688 scope.go:122] "RemoveContainer" containerID="09cb46df9108ad90ba389b2a4d4f9ac487078bb2bba5095bddab888eb3a8afc0" Oct 31 19:00:48.450111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.450077 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn"] Oct 31 19:00:48.454640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.454608 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-794d786ff5-kpskn"] Oct 31 19:00:48.864796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.864765 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj"] Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865157 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865173 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44d7d5d-926e-46ca-836f-76b5dc7fa009" containerName="cleanup" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865258 5688 cpu_manager.go:414] "containerMap: removing container" podUID="03808518-b1bf-4e9e-bdfd-e49dafc93021" containerName="payload-pre-processing" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865265 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03808518-b1bf-4e9e-bdfd-e49dafc93021" containerName="payload-pre-processing" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865274 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" containerName="maas-api" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865279 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" containerName="maas-api" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865296 5688 cpu_manager.go:414] "containerMap: removing container" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:48.865330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865301 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865401 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c330ecb-c9ce-40dc-8ef5-28f708a0ad64" containerName="maas-api" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865411 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cd622b-ad09-459d-8c9e-7f0c76e5be24" containerName="cleanup" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865419 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="03808518-b1bf-4e9e-bdfd-e49dafc93021" containerName="payload-pre-processing" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865426 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e56fc5e-eca7-44a8-9824-0e1fbc8ac1b9" containerName="cleanup" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865435 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebc3615-4215-4b83-8909-93d05364df57" containerName="cleanup" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865441 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4ee0d5-d505-428e-a6b7-479af7c9f35f" containerName="cleanup" Oct 31 19:00:48.865726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.865448 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdda03cf-d2d9-4a8b-8dc4-c49b84474913" containerName="cleanup" Oct 31 19:00:48.869880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.869848 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:48.930665 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.930636 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4c58c5-dockercfg-tbrkk\"" Oct 31 19:00:48.931043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.931024 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4c58c5\"" Oct 31 19:00:48.936244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.936221 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj"] Oct 31 19:00:48.940675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.940646 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:48.940826 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:48.940800 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wtz\" (UniqueName: \"kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz\") pod \"payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:49.015564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.015533 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb"] Oct 31 19:00:49.019516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.019492 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.041501 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.041473 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzlh\" (UniqueName: \"kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh\") pod \"payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.041630 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.041521 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.041630 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.041588 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:49.041736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.041659 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wtz\" (UniqueName: \"kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz\") pod \"payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:49.042221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.042177 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:49.076153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.076128 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb"] Oct 31 19:00:49.097842 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.097819 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wtz\" (UniqueName: \"kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz\") pod \"payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:49.143028 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.142965 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.143144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.143132 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzlh\" (UniqueName: \"kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh\") pod \"payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.143643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.143625 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.182220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.182196 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:49.199724 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.199703 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzlh\" (UniqueName: \"kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh\") pod \"payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.332678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.332647 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:49.562457 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:49.562422 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4938e46_aa53_49cd_bc2c_ba916d4e6077.slice/crio-a414357460948d2de698a859ebdf1efb84479be3ddc21ed775a54fd03d923148 WatchSource:0}: Error finding container a414357460948d2de698a859ebdf1efb84479be3ddc21ed775a54fd03d923148: Status 404 returned error can't find the container with id a414357460948d2de698a859ebdf1efb84479be3ddc21ed775a54fd03d923148 Oct 31 19:00:49.567789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.567765 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03808518-b1bf-4e9e-bdfd-e49dafc93021" path="/var/lib/kubelet/pods/03808518-b1bf-4e9e-bdfd-e49dafc93021/volumes" Oct 31 19:00:49.568096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.568078 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj"] Oct 31 19:00:49.712149 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:49.712110 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb"] Oct 31 19:00:49.714684 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:00:49.714659 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273a43c7_34c0_4fea_8684_f598407e8aba.slice/crio-5e5904de13847523bf9ee7283939d6a3625306cb82b1895fc508d9e8aba4f31f WatchSource:0}: Error finding container 5e5904de13847523bf9ee7283939d6a3625306cb82b1895fc508d9e8aba4f31f: Status 404 returned error can't find the container with id 5e5904de13847523bf9ee7283939d6a3625306cb82b1895fc508d9e8aba4f31f Oct 31 19:00:50.299769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.299731 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" event={"ID":"273a43c7-34c0-4fea-8684-f598407e8aba","Type":"ContainerStarted","Data":"8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d"} Oct 31 19:00:50.300262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.299952 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:00:50.300262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.299993 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" event={"ID":"273a43c7-34c0-4fea-8684-f598407e8aba","Type":"ContainerStarted","Data":"5e5904de13847523bf9ee7283939d6a3625306cb82b1895fc508d9e8aba4f31f"} Oct 31 19:00:50.301350 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.301322 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" event={"ID":"e4938e46-aa53-49cd-bc2c-ba916d4e6077","Type":"ContainerStarted","Data":"8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4"} Oct 31 19:00:50.301453 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.301365 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" event={"ID":"e4938e46-aa53-49cd-bc2c-ba916d4e6077","Type":"ContainerStarted","Data":"a414357460948d2de698a859ebdf1efb84479be3ddc21ed775a54fd03d923148"} Oct 31 19:00:50.301453 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.301405 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:00:50.413639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.413594 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" podStartSLOduration=2.413584176 podStartE2EDuration="2.413584176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:50.413031243 +0000 UTC m=+2371.015172395" watchObservedRunningTime="2026-10-31 19:00:50.413584176 +0000 UTC m=+2371.015725319" Oct 31 19:00:50.540650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:50.528021 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" podStartSLOduration=2.528003464 podStartE2EDuration="2.528003464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:50.527453887 +0000 UTC m=+2371.129595034" watchObservedRunningTime="2026-10-31 19:00:50.528003464 +0000 UTC m=+2371.130144603" Oct 31 19:00:54.904372 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:00:54.904335 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:00:57.900558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:57.900523 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv"] Oct 31 19:00:57.901026 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:00:57.900778 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" containerID="cri-o://dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a" gracePeriod=30 Oct 31 19:01:00.795157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.795123 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc"] Oct 31 19:01:00.799602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.799578 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.878528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.872838 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.878528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.872902 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv69\" (UniqueName: \"kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.878528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.872930 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.902786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.902759 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:01:00.902995 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.902976 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:01:00.904746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.904722 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc"] Oct 31 19:01:00.973688 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.973667 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.973813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.973702 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv69\" (UniqueName: \"kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.973813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.973722 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.974340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.974313 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:00.976208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:00.976170 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:01.030216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:01.030176 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv69\" (UniqueName: \"kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69\") pod \"maas-api-revoke-sub-5dea0bdd-7vwnc\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:01.114042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:01.113980 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:01.309484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:01.309455 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:01:01.309621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:01.309521 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:01:01.536567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:01.536544 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc"] Oct 31 19:01:01.538950 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:01:01.538915 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776c0213_a96d_4d2a_850c_a3917e5bc1e9.slice/crio-5ca2d589cbec30174c3944e5370f8102256910702047097e3238ac31b249054f WatchSource:0}: Error finding container 5ca2d589cbec30174c3944e5370f8102256910702047097e3238ac31b249054f: Status 404 returned error can't find the container with id 5ca2d589cbec30174c3944e5370f8102256910702047097e3238ac31b249054f Oct 31 19:01:02.379634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:02.379594 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" event={"ID":"776c0213-a96d-4d2a-850c-a3917e5bc1e9","Type":"ContainerStarted","Data":"ef41ac41c43f6a1b65e52e0e2b19a3840a9576287ce62c43025cf0728a898373"} Oct 31 19:01:02.380055 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:02.379663 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" event={"ID":"776c0213-a96d-4d2a-850c-a3917e5bc1e9","Type":"ContainerStarted","Data":"5ca2d589cbec30174c3944e5370f8102256910702047097e3238ac31b249054f"} Oct 31 19:01:02.494588 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:02.494547 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" podStartSLOduration=2.494534198 podStartE2EDuration="2.494534198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:01:02.492281811 +0000 UTC m=+2383.094422954" watchObservedRunningTime="2026-10-31 19:01:02.494534198 +0000 UTC m=+2383.096675580" Oct 31 19:01:03.145406 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.145384 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 19:01:03.194911 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.194840 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.194911 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.194887 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.194939 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.194969 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195060 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195120 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195149 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195239 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195276 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data\") pod \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\" (UID: \"bbf67da8-9280-46ed-8f9a-0354df5fc2e7\") " Oct 31 19:01:03.195514 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195115 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:01:03.195514 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195471 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:01:03.195831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195692 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:01:03.195831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195807 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.195831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195829 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.196038 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195832 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:01:03.196038 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195843 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.196038 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.195942 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:01:03.197778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.197752 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:01:03.198511 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.198487 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:03.198511 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.198499 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "kube-api-access-c9gcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:03.198703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.198673 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy" pod "bbf67da8-9280-46ed-8f9a-0354df5fc2e7" (UID: "bbf67da8-9280-46ed-8f9a-0354df5fc2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:01:03.296633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.296603 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.296633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.296632 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gcc\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-kube-api-access-c9gcc\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.296795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.296641 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.296795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.296649 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.296795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.296658 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.296795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.296666 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf67da8-9280-46ed-8f9a-0354df5fc2e7-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:03.387841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.387811 5688 generic.go:356] "Generic (PLEG): container finished" podID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerID="dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a" exitCode=0 Oct 31 19:01:03.388320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.387873 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" Oct 31 19:01:03.388320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.387905 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" event={"ID":"bbf67da8-9280-46ed-8f9a-0354df5fc2e7","Type":"ContainerDied","Data":"dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a"} Oct 31 19:01:03.388320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.387959 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv" event={"ID":"bbf67da8-9280-46ed-8f9a-0354df5fc2e7","Type":"ContainerDied","Data":"02f74b0e5025a7664f57bf4bb34af4788bc65b0a136f2efec2a0ec2a8ddebcdf"} Oct 31 19:01:03.388320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.387988 5688 scope.go:122] "RemoveContainer" containerID="dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a" Oct 31 19:01:03.389978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.389838 5688 generic.go:356] "Generic (PLEG): container finished" podID="776c0213-a96d-4d2a-850c-a3917e5bc1e9" containerID="ef41ac41c43f6a1b65e52e0e2b19a3840a9576287ce62c43025cf0728a898373" exitCode=0 Oct 31 19:01:03.389978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.389930 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" event={"ID":"776c0213-a96d-4d2a-850c-a3917e5bc1e9","Type":"ContainerDied","Data":"ef41ac41c43f6a1b65e52e0e2b19a3840a9576287ce62c43025cf0728a898373"} Oct 31 19:01:03.400141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.400115 5688 scope.go:122] "RemoveContainer" containerID="f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0" Oct 31 19:01:03.410497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.410475 5688 scope.go:122] "RemoveContainer" containerID="dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a" Oct 31 19:01:03.410767 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:01:03.410749 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a\": container with ID starting with dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a not found: ID does not exist" containerID="dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a" Oct 31 19:01:03.410824 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.410774 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a"} err="failed to get container status \"dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a\": rpc error: code = NotFound desc = could not find container \"dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a\": container with ID starting with dd4d6cbaaf890f39e20785134e24e5d575d6ca19233d0980b8a8508e3566ea3a not found: ID does not exist" Oct 31 19:01:03.410824 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.410793 5688 scope.go:122] "RemoveContainer" containerID="f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0" Oct 31 19:01:03.411546 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:01:03.411524 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0\": container with ID starting with f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0 not found: ID does not exist" containerID="f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0" Oct 31 19:01:03.411619 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.411554 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0"} err="failed to get container status \"f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0\": rpc error: code = NotFound desc = could not find container \"f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0\": container with ID starting with f7d338f9c510fe40e0b3bf154a976cfe00279aec76fdb4fce8cbcdac5400bbf0 not found: ID does not exist" Oct 31 19:01:03.551053 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.551025 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv"] Oct 31 19:01:03.555174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.555151 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-676224-openshift-default-97d7c7797-5lrqv"] Oct 31 19:01:03.566356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:03.566327 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" path="/var/lib/kubelet/pods/bbf67da8-9280-46ed-8f9a-0354df5fc2e7/volumes" Oct 31 19:01:04.545929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.545899 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:04.608769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.608732 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca\") pod \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " Oct 31 19:01:04.608905 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.608809 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca\") pod \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " Oct 31 19:01:04.609028 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.609013 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69\" (UniqueName: \"kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69\") pod \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\" (UID: \"776c0213-a96d-4d2a-850c-a3917e5bc1e9\") " Oct 31 19:01:04.609115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.609094 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca" pod "776c0213-a96d-4d2a-850c-a3917e5bc1e9" (UID: "776c0213-a96d-4d2a-850c-a3917e5bc1e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:01:04.609403 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.609382 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:04.610834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.610813 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca" pod "776c0213-a96d-4d2a-850c-a3917e5bc1e9" (UID: "776c0213-a96d-4d2a-850c-a3917e5bc1e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:01:04.611035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.611016 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69" pod "776c0213-a96d-4d2a-850c-a3917e5bc1e9" (UID: "776c0213-a96d-4d2a-850c-a3917e5bc1e9"). InnerVolumeSpecName "kube-api-access-4zv69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:04.710379 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.710299 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zv69\" (UniqueName: \"kubernetes.io/projected/776c0213-a96d-4d2a-850c-a3917e5bc1e9-kube-api-access-4zv69\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:04.710379 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:04.710338 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c0213-a96d-4d2a-850c-a3917e5bc1e9-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:05.404072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:05.404043 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" Oct 31 19:01:05.404297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:05.404046 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc" event={"ID":"776c0213-a96d-4d2a-850c-a3917e5bc1e9","Type":"ContainerDied","Data":"5ca2d589cbec30174c3944e5370f8102256910702047097e3238ac31b249054f"} Oct 31 19:01:05.404297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:05.404164 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca2d589cbec30174c3944e5370f8102256910702047097e3238ac31b249054f" Oct 31 19:01:08.616041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.616008 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6"] Oct 31 19:01:08.616489 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.616229 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" podUID="0d4d1022-042d-432b-bae4-43ab46a7d6aa" containerName="payload-processing" containerID="cri-o://ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591" gracePeriod=30 Oct 31 19:01:08.873915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.873857 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 19:01:08.949842 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.949811 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8\" (UniqueName: \"kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8\") pod \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " Oct 31 19:01:08.949980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.949956 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume\") pod \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\" (UID: \"0d4d1022-042d-432b-bae4-43ab46a7d6aa\") " Oct 31 19:01:08.950370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.950337 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume" pod "0d4d1022-042d-432b-bae4-43ab46a7d6aa" (UID: "0d4d1022-042d-432b-bae4-43ab46a7d6aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:01:08.952008 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:08.951986 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8" pod "0d4d1022-042d-432b-bae4-43ab46a7d6aa" (UID: "0d4d1022-042d-432b-bae4-43ab46a7d6aa"). InnerVolumeSpecName "kube-api-access-pztn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:09.051395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.051367 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d1022-042d-432b-bae4-43ab46a7d6aa-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:09.051395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.051390 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztn8\" (UniqueName: \"kubernetes.io/projected/0d4d1022-042d-432b-bae4-43ab46a7d6aa-kube-api-access-pztn8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:09.424595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.424559 5688 generic.go:356] "Generic (PLEG): container finished" podID="0d4d1022-042d-432b-bae4-43ab46a7d6aa" containerID="ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591" exitCode=0 Oct 31 19:01:09.424788 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.424607 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" event={"ID":"0d4d1022-042d-432b-bae4-43ab46a7d6aa","Type":"ContainerDied","Data":"ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591"} Oct 31 19:01:09.424788 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.424685 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" event={"ID":"0d4d1022-042d-432b-bae4-43ab46a7d6aa","Type":"ContainerDied","Data":"451b74e4789ef6fcdc46326a52dd254a211314d25759e537459c8e894aeed619"} Oct 31 19:01:09.424788 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.424619 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6" Oct 31 19:01:09.424788 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.424707 5688 scope.go:122] "RemoveContainer" containerID="ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591" Oct 31 19:01:09.449704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.442378 5688 scope.go:122] "RemoveContainer" containerID="ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591" Oct 31 19:01:09.453799 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:01:09.453759 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591\": container with ID starting with ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591 not found: ID does not exist" containerID="ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591" Oct 31 19:01:09.454074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.454030 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591"} err="failed to get container status \"ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591\": rpc error: code = NotFound desc = could not find container \"ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591\": container with ID starting with ca0195331f8914bd8405934c2bea5121790d74f712443f0f00e61eb75c9df591 not found: ID does not exist" Oct 31 19:01:09.587068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.587038 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6"] Oct 31 19:01:09.592909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:09.592886 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-6c9cf48776-5d7t6"] Oct 31 19:01:11.568544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:11.568513 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4d1022-042d-432b-bae4-43ab46a7d6aa" path="/var/lib/kubelet/pods/0d4d1022-042d-432b-bae4-43ab46a7d6aa/volumes" Oct 31 19:01:14.440427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.440392 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7"] Oct 31 19:01:14.441052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441030 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0d4d1022-042d-432b-bae4-43ab46a7d6aa" containerName="payload-processing" Oct 31 19:01:14.441166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441054 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4d1022-042d-432b-bae4-43ab46a7d6aa" containerName="payload-processing" Oct 31 19:01:14.441166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441069 5688 cpu_manager.go:414] "containerMap: removing container" podUID="776c0213-a96d-4d2a-850c-a3917e5bc1e9" containerName="revoke-keys" Oct 31 19:01:14.441166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441078 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776c0213-a96d-4d2a-850c-a3917e5bc1e9" containerName="revoke-keys" Oct 31 19:01:14.441166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441113 5688 cpu_manager.go:414] "containerMap: removing container" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" Oct 31 19:01:14.441166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441122 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" Oct 31 19:01:14.441369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441168 5688 cpu_manager.go:414] "containerMap: removing container" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" Oct 31 19:01:14.441369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441175 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" Oct 31 19:01:14.441369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441324 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" Oct 31 19:01:14.441369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441336 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf67da8-9280-46ed-8f9a-0354df5fc2e7" containerName="istio-proxy" Oct 31 19:01:14.441369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441349 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4d1022-042d-432b-bae4-43ab46a7d6aa" containerName="payload-processing" Oct 31 19:01:14.441369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.441359 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="776c0213-a96d-4d2a-850c-a3917e5bc1e9" containerName="revoke-keys" Oct 31 19:01:14.445069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.445048 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509410 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509381 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509552 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509441 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhtk\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509552 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509523 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509629 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509556 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509629 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509583 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509669 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509737 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509721 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509771 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509753 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.509803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.509776 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.533959 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.533933 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7"] Oct 31 19:01:14.546935 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.546913 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-79e7b6-openshift-default-dockercfg-495g2\"" Oct 31 19:01:14.611426 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.611384 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.611764 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.611741 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.611937 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.611921 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhtk\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.612075 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.612062 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.612223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.612205 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.618601 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.612351 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.618882 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.618856 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.619000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.618937 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.619042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.619028 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.622465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.622437 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.622765 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.622740 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.622973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.622954 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.623077 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.623010 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.623751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.623723 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.623988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.623965 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.624470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.624442 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.688637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.688289 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhtk\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.688829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.688680 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token\") pod \"e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:14.761101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:14.761071 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:15.197988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.197953 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7"] Oct 31 19:01:15.200862 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:01:15.200832 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169b7f77_ea2e_4d64_925c_eb082102aa78.slice/crio-a33ac931f096eda4eeb0593fc9500b288cc6f0dc9d790c5197a9e2ad6fc02acf WatchSource:0}: Error finding container a33ac931f096eda4eeb0593fc9500b288cc6f0dc9d790c5197a9e2ad6fc02acf: Status 404 returned error can't find the container with id a33ac931f096eda4eeb0593fc9500b288cc6f0dc9d790c5197a9e2ad6fc02acf Oct 31 19:01:15.203811 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.203766 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:01:15.203932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.203863 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:01:15.203932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.203912 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:01:15.460615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.460529 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" event={"ID":"169b7f77-ea2e-4d64-925c-eb082102aa78","Type":"ContainerStarted","Data":"2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3"} Oct 31 19:01:15.460615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.460591 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" event={"ID":"169b7f77-ea2e-4d64-925c-eb082102aa78","Type":"ContainerStarted","Data":"a33ac931f096eda4eeb0593fc9500b288cc6f0dc9d790c5197a9e2ad6fc02acf"} Oct 31 19:01:15.585535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.585477 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podStartSLOduration=1.585456863 podStartE2EDuration="1.585456863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:01:15.582942706 +0000 UTC m=+2396.185083862" watchObservedRunningTime="2026-10-31 19:01:15.585456863 +0000 UTC m=+2396.187598009" Oct 31 19:01:15.762154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:15.762123 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:16.767763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.767736 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:16.815845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.815814 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp"] Oct 31 19:01:16.819426 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.819405 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:16.874623 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.874602 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:01:16.875088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.875047 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:01:16.876950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.876927 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp"] Oct 31 19:01:16.942683 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.942655 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:16.942808 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.942685 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:16.942808 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:16.942728 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcvm\" (UniqueName: \"kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.044126 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.044069 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.044126 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.044098 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.044355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.044143 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcvm\" (UniqueName: \"kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.044689 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.044669 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.046556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.046533 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.101408 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.101376 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcvm\" (UniqueName: \"kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm\") pod \"maas-api-revoke-sub-e7a453a2-d7dxp\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.132986 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.132957 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:17.477820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.477708 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:17.479115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.479090 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:01:17.504923 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:17.504901 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp"] Oct 31 19:01:17.507020 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:01:17.506989 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27371528_d4b7_402d_9a85_5cd67aa08af0.slice/crio-a3bb02d02fb46ec2a9222943105705abfacc752afd33e59c34da77fc342ccd06 WatchSource:0}: Error finding container a3bb02d02fb46ec2a9222943105705abfacc752afd33e59c34da77fc342ccd06: Status 404 returned error can't find the container with id a3bb02d02fb46ec2a9222943105705abfacc752afd33e59c34da77fc342ccd06 Oct 31 19:01:18.482284 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:18.482249 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" event={"ID":"27371528-d4b7-402d-9a85-5cd67aa08af0","Type":"ContainerStarted","Data":"9bc92b27dfaf901a586c9d1e02a2b7f47084472f43cf9c1bd442e82c82f51406"} Oct 31 19:01:18.482722 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:18.482306 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" event={"ID":"27371528-d4b7-402d-9a85-5cd67aa08af0","Type":"ContainerStarted","Data":"a3bb02d02fb46ec2a9222943105705abfacc752afd33e59c34da77fc342ccd06"} Oct 31 19:01:18.603944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:18.603896 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" podStartSLOduration=2.603881969 podStartE2EDuration="2.603881969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:01:18.602711909 +0000 UTC m=+2399.204853244" watchObservedRunningTime="2026-10-31 19:01:18.603881969 +0000 UTC m=+2399.206023232" Oct 31 19:01:19.489453 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:19.489421 5688 generic.go:356] "Generic (PLEG): container finished" podID="27371528-d4b7-402d-9a85-5cd67aa08af0" containerID="9bc92b27dfaf901a586c9d1e02a2b7f47084472f43cf9c1bd442e82c82f51406" exitCode=0 Oct 31 19:01:19.489929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:19.489503 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" event={"ID":"27371528-d4b7-402d-9a85-5cd67aa08af0","Type":"ContainerDied","Data":"9bc92b27dfaf901a586c9d1e02a2b7f47084472f43cf9c1bd442e82c82f51406"} Oct 31 19:01:19.891620 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:19.891538 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:01:19.894278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:19.894251 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:01:20.630417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.630391 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:20.779382 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.779355 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca\") pod \"27371528-d4b7-402d-9a85-5cd67aa08af0\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " Oct 31 19:01:20.779482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.779423 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca\") pod \"27371528-d4b7-402d-9a85-5cd67aa08af0\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " Oct 31 19:01:20.779630 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.779615 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm\" (UniqueName: \"kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm\") pod \"27371528-d4b7-402d-9a85-5cd67aa08af0\" (UID: \"27371528-d4b7-402d-9a85-5cd67aa08af0\") " Oct 31 19:01:20.780016 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.779846 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca" pod "27371528-d4b7-402d-9a85-5cd67aa08af0" (UID: "27371528-d4b7-402d-9a85-5cd67aa08af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:01:20.782216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.782155 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm" pod "27371528-d4b7-402d-9a85-5cd67aa08af0" (UID: "27371528-d4b7-402d-9a85-5cd67aa08af0"). InnerVolumeSpecName "kube-api-access-2xcvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:20.782216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.782176 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca" pod "27371528-d4b7-402d-9a85-5cd67aa08af0" (UID: "27371528-d4b7-402d-9a85-5cd67aa08af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:01:20.881119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.881093 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:20.881119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.881115 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27371528-d4b7-402d-9a85-5cd67aa08af0-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:20.881298 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:20.881126 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xcvm\" (UniqueName: \"kubernetes.io/projected/27371528-d4b7-402d-9a85-5cd67aa08af0-kube-api-access-2xcvm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:21.501236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:21.501207 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" Oct 31 19:01:21.501236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:21.501220 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp" event={"ID":"27371528-d4b7-402d-9a85-5cd67aa08af0","Type":"ContainerDied","Data":"a3bb02d02fb46ec2a9222943105705abfacc752afd33e59c34da77fc342ccd06"} Oct 31 19:01:21.501434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:21.501296 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3bb02d02fb46ec2a9222943105705abfacc752afd33e59c34da77fc342ccd06" Oct 31 19:01:23.435655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.435623 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq"] Oct 31 19:01:23.436041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.436029 5688 cpu_manager.go:414] "containerMap: removing container" podUID="27371528-d4b7-402d-9a85-5cd67aa08af0" containerName="revoke-keys" Oct 31 19:01:23.436088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.436041 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27371528-d4b7-402d-9a85-5cd67aa08af0" containerName="revoke-keys" Oct 31 19:01:23.436144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.436135 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="27371528-d4b7-402d-9a85-5cd67aa08af0" containerName="revoke-keys" Oct 31 19:01:23.440490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.440475 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.497007 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.496978 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq"] Oct 31 19:01:23.513896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.510579 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.513896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.510761 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkz9\" (UniqueName: \"kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9\") pod \"payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.546498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.544416 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-79e7b6\"" Oct 31 19:01:23.546498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.544542 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-79e7b6-dockercfg-5kjqb\"" Oct 31 19:01:23.612040 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.612010 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkz9\" (UniqueName: \"kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9\") pod \"payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.612280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.612089 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.612782 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.612763 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.668459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.668424 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkz9\" (UniqueName: \"kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9\") pod \"payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:23.752872 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:23.752848 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:24.126034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:24.125863 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq"] Oct 31 19:01:24.128663 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:01:24.128637 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8fb434_3b1c_4ab8_b600_3f42a319436b.slice/crio-1c5632a345404b4a4ce676dd23afef27151f6826fe0089c7c98f3f368487719e WatchSource:0}: Error finding container 1c5632a345404b4a4ce676dd23afef27151f6826fe0089c7c98f3f368487719e: Status 404 returned error can't find the container with id 1c5632a345404b4a4ce676dd23afef27151f6826fe0089c7c98f3f368487719e Oct 31 19:01:24.542818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:24.542785 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" event={"ID":"1f8fb434-3b1c-4ab8-b600-3f42a319436b","Type":"ContainerStarted","Data":"dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9"} Oct 31 19:01:24.543247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:24.542832 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" event={"ID":"1f8fb434-3b1c-4ab8-b600-3f42a319436b","Type":"ContainerStarted","Data":"1c5632a345404b4a4ce676dd23afef27151f6826fe0089c7c98f3f368487719e"} Oct 31 19:01:24.543247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:24.542901 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:24.654979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:24.654927 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" podStartSLOduration=1.654915351 podStartE2EDuration="1.654915351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:01:24.65473481 +0000 UTC m=+2405.256875953" watchObservedRunningTime="2026-10-31 19:01:24.654915351 +0000 UTC m=+2405.257056495" Oct 31 19:01:24.904245 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:01:24.904135 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:01:32.763220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.763173 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8"] Oct 31 19:01:32.767049 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.767034 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:32.822956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.822930 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8"] Oct 31 19:01:32.870014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.869977 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:01:32.870158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.870069 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:01:32.902955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.902930 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmshv\" (UniqueName: \"kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:32.903056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.902980 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:32.903097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:32.903080 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.003763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.003734 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.003918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.003827 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmshv\" (UniqueName: \"kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.003918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.003893 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.004529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.004501 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.006255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.006238 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.059652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.059596 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmshv\" (UniqueName: \"kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv\") pod \"maas-api-revoke-sub-3d93f588-2p9j8\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.079870 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.079847 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:33.458211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.458155 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8"] Oct 31 19:01:33.460377 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:01:33.460349 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9df0e23_eb88_4ac4_8d89_744eece81be5.slice/crio-2ebfd495ae859ad585887f45845c8966c623d0020044b068192f9397f51be56f WatchSource:0}: Error finding container 2ebfd495ae859ad585887f45845c8966c623d0020044b068192f9397f51be56f: Status 404 returned error can't find the container with id 2ebfd495ae859ad585887f45845c8966c623d0020044b068192f9397f51be56f Oct 31 19:01:33.587331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:33.587300 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" event={"ID":"f9df0e23-eb88-4ac4-8d89-744eece81be5","Type":"ContainerStarted","Data":"2ebfd495ae859ad585887f45845c8966c623d0020044b068192f9397f51be56f"} Oct 31 19:01:34.593815 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:34.593743 5688 generic.go:356] "Generic (PLEG): container finished" podID="f9df0e23-eb88-4ac4-8d89-744eece81be5" containerID="dfa2c6fd94472092f59656bc7bcc70ab6908da069048a43038527523091d4f12" exitCode=0 Oct 31 19:01:34.594154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:34.593834 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" event={"ID":"f9df0e23-eb88-4ac4-8d89-744eece81be5","Type":"ContainerDied","Data":"dfa2c6fd94472092f59656bc7bcc70ab6908da069048a43038527523091d4f12"} Oct 31 19:01:35.550852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.550824 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:01:35.747308 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.747286 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:35.931768 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.931693 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca\") pod \"f9df0e23-eb88-4ac4-8d89-744eece81be5\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " Oct 31 19:01:35.931938 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.931788 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca\") pod \"f9df0e23-eb88-4ac4-8d89-744eece81be5\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " Oct 31 19:01:35.931938 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.931836 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv\" (UniqueName: \"kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv\") pod \"f9df0e23-eb88-4ac4-8d89-744eece81be5\" (UID: \"f9df0e23-eb88-4ac4-8d89-744eece81be5\") " Oct 31 19:01:35.932143 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.932122 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca" pod "f9df0e23-eb88-4ac4-8d89-744eece81be5" (UID: "f9df0e23-eb88-4ac4-8d89-744eece81be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:01:35.932240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.932228 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:35.933934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.933911 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca" pod "f9df0e23-eb88-4ac4-8d89-744eece81be5" (UID: "f9df0e23-eb88-4ac4-8d89-744eece81be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:01:35.933998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:35.933965 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv" pod "f9df0e23-eb88-4ac4-8d89-744eece81be5" (UID: "f9df0e23-eb88-4ac4-8d89-744eece81be5"). InnerVolumeSpecName "kube-api-access-nmshv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:36.033174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:36.033147 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9df0e23-eb88-4ac4-8d89-744eece81be5-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:36.033330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:36.033175 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmshv\" (UniqueName: \"kubernetes.io/projected/f9df0e23-eb88-4ac4-8d89-744eece81be5-kube-api-access-nmshv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:36.606605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:36.606574 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" Oct 31 19:01:36.606780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:36.606609 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8" event={"ID":"f9df0e23-eb88-4ac4-8d89-744eece81be5","Type":"ContainerDied","Data":"2ebfd495ae859ad585887f45845c8966c623d0020044b068192f9397f51be56f"} Oct 31 19:01:36.606780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:36.606656 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ebfd495ae859ad585887f45845c8966c623d0020044b068192f9397f51be56f" Oct 31 19:01:48.948538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:48.948501 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk"] Oct 31 19:01:48.949236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:48.949215 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f9df0e23-eb88-4ac4-8d89-744eece81be5" containerName="revoke-keys" Oct 31 19:01:48.949309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:48.949238 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9df0e23-eb88-4ac4-8d89-744eece81be5" containerName="revoke-keys" Oct 31 19:01:48.949358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:48.949348 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9df0e23-eb88-4ac4-8d89-744eece81be5" containerName="revoke-keys" Oct 31 19:01:48.952932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:48.952911 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.023893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.023868 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk"] Oct 31 19:01:49.057395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.057374 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:01:49.057395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.057375 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:01:49.064844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.064823 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kr7\" (UniqueName: \"kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.064937 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.064911 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.065059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.065038 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.165734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.165693 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.165926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.165830 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.165926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.165863 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kr7\" (UniqueName: \"kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.168222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.167441 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.169377 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.169350 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.222551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.222483 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kr7\" (UniqueName: \"kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7\") pod \"maas-api-revoke-sub-40fca44f-bgkpk\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.266042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.266014 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:49.639804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.639783 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk"] Oct 31 19:01:49.642559 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:01:49.642533 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1c9eda_c202_44da_bdfd_5a16419476bf.slice/crio-d9efed5e7294f509160cdb6c6bc1ca8c2f28ce26a5b73341f0509c8b2ac44936 WatchSource:0}: Error finding container d9efed5e7294f509160cdb6c6bc1ca8c2f28ce26a5b73341f0509c8b2ac44936: Status 404 returned error can't find the container with id d9efed5e7294f509160cdb6c6bc1ca8c2f28ce26a5b73341f0509c8b2ac44936 Oct 31 19:01:49.672014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:49.671983 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" event={"ID":"eb1c9eda-c202-44da-bdfd-5a16419476bf","Type":"ContainerStarted","Data":"d9efed5e7294f509160cdb6c6bc1ca8c2f28ce26a5b73341f0509c8b2ac44936"} Oct 31 19:01:50.678756 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:50.678719 5688 generic.go:356] "Generic (PLEG): container finished" podID="eb1c9eda-c202-44da-bdfd-5a16419476bf" containerID="3c77e919e58c834554843174c9f2508fb070315403083da175e2aec67cdf977f" exitCode=0 Oct 31 19:01:50.679178 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:50.678775 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" event={"ID":"eb1c9eda-c202-44da-bdfd-5a16419476bf","Type":"ContainerDied","Data":"3c77e919e58c834554843174c9f2508fb070315403083da175e2aec67cdf977f"} Oct 31 19:01:51.843459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.843432 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:51.893320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.893294 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca\") pod \"eb1c9eda-c202-44da-bdfd-5a16419476bf\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " Oct 31 19:01:51.893473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.893372 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca\") pod \"eb1c9eda-c202-44da-bdfd-5a16419476bf\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " Oct 31 19:01:51.893538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.893478 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7\" (UniqueName: \"kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7\") pod \"eb1c9eda-c202-44da-bdfd-5a16419476bf\" (UID: \"eb1c9eda-c202-44da-bdfd-5a16419476bf\") " Oct 31 19:01:51.893704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.893681 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca" pod "eb1c9eda-c202-44da-bdfd-5a16419476bf" (UID: "eb1c9eda-c202-44da-bdfd-5a16419476bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:01:51.893827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.893806 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:51.895476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.895450 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca" pod "eb1c9eda-c202-44da-bdfd-5a16419476bf" (UID: "eb1c9eda-c202-44da-bdfd-5a16419476bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:01:51.895687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.895664 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7" pod "eb1c9eda-c202-44da-bdfd-5a16419476bf" (UID: "eb1c9eda-c202-44da-bdfd-5a16419476bf"). InnerVolumeSpecName "kube-api-access-d2kr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:01:51.995153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.995124 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kr7\" (UniqueName: \"kubernetes.io/projected/eb1c9eda-c202-44da-bdfd-5a16419476bf-kube-api-access-d2kr7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:51.995153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:51.995150 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1c9eda-c202-44da-bdfd-5a16419476bf-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:01:52.689858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:52.689824 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" event={"ID":"eb1c9eda-c202-44da-bdfd-5a16419476bf","Type":"ContainerDied","Data":"d9efed5e7294f509160cdb6c6bc1ca8c2f28ce26a5b73341f0509c8b2ac44936"} Oct 31 19:01:52.689858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:52.689843 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk" Oct 31 19:01:52.690073 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:01:52.689896 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9efed5e7294f509160cdb6c6bc1ca8c2f28ce26a5b73341f0509c8b2ac44936" Oct 31 19:01:54.903351 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:01:54.903316 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:02:04.872476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.872405 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt"] Oct 31 19:02:04.872956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.872829 5688 cpu_manager.go:414] "containerMap: removing container" podUID="eb1c9eda-c202-44da-bdfd-5a16419476bf" containerName="revoke-keys" Oct 31 19:02:04.872956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.872840 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1c9eda-c202-44da-bdfd-5a16419476bf" containerName="revoke-keys" Oct 31 19:02:04.872956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.872922 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1c9eda-c202-44da-bdfd-5a16419476bf" containerName="revoke-keys" Oct 31 19:02:04.877234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.877220 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:04.980793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.980764 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:02:04.981042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.981022 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:02:04.982997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:04.982976 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt"] Oct 31 19:02:05.036259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.036235 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.036536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.036516 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9g6\" (UniqueName: \"kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.036714 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.036701 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.138212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.138129 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.138318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.138232 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.138318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.138265 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9g6\" (UniqueName: \"kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.138708 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.138688 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.140625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.140599 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.195199 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.195158 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9g6\" (UniqueName: \"kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6\") pod \"maas-api-revoke-sub-93ce059f-tm5bt\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.489595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.489567 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:05.860725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:05.860612 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt"] Oct 31 19:02:05.863040 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:02:05.863013 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3972e76b_9f4e_4039_a847_9caf4eb841fe.slice/crio-5e9d4163e7bc0d53c5e58a7076314306e4676cd8ab74c9298eb3b2266214bbf3 WatchSource:0}: Error finding container 5e9d4163e7bc0d53c5e58a7076314306e4676cd8ab74c9298eb3b2266214bbf3: Status 404 returned error can't find the container with id 5e9d4163e7bc0d53c5e58a7076314306e4676cd8ab74c9298eb3b2266214bbf3 Oct 31 19:02:06.765536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:06.765504 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" event={"ID":"3972e76b-9f4e-4039-a847-9caf4eb841fe","Type":"ContainerStarted","Data":"93f47c1c3461f60dfdb974bc190a36e7c6cf6687e61fdf63b84bab0e3c57f982"} Oct 31 19:02:06.765906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:06.765558 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" event={"ID":"3972e76b-9f4e-4039-a847-9caf4eb841fe","Type":"ContainerStarted","Data":"5e9d4163e7bc0d53c5e58a7076314306e4676cd8ab74c9298eb3b2266214bbf3"} Oct 31 19:02:06.878236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:06.878158 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" podStartSLOduration=2.878145127 podStartE2EDuration="2.878145127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:06.87793493 +0000 UTC m=+2447.480076082" watchObservedRunningTime="2026-10-31 19:02:06.878145127 +0000 UTC m=+2447.480286267" Oct 31 19:02:07.771848 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:07.771814 5688 generic.go:356] "Generic (PLEG): container finished" podID="3972e76b-9f4e-4039-a847-9caf4eb841fe" containerID="93f47c1c3461f60dfdb974bc190a36e7c6cf6687e61fdf63b84bab0e3c57f982" exitCode=0 Oct 31 19:02:07.772291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:07.771902 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" event={"ID":"3972e76b-9f4e-4039-a847-9caf4eb841fe","Type":"ContainerDied","Data":"93f47c1c3461f60dfdb974bc190a36e7c6cf6687e61fdf63b84bab0e3c57f982"} Oct 31 19:02:08.934064 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:08.934042 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:09.078008 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.077931 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6\" (UniqueName: \"kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6\") pod \"3972e76b-9f4e-4039-a847-9caf4eb841fe\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " Oct 31 19:02:09.078008 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.077991 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca\") pod \"3972e76b-9f4e-4039-a847-9caf4eb841fe\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " Oct 31 19:02:09.078268 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.078083 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca\") pod \"3972e76b-9f4e-4039-a847-9caf4eb841fe\" (UID: \"3972e76b-9f4e-4039-a847-9caf4eb841fe\") " Oct 31 19:02:09.078864 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.078787 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca" pod "3972e76b-9f4e-4039-a847-9caf4eb841fe" (UID: "3972e76b-9f4e-4039-a847-9caf4eb841fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:02:09.080346 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.080323 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca" pod "3972e76b-9f4e-4039-a847-9caf4eb841fe" (UID: "3972e76b-9f4e-4039-a847-9caf4eb841fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:02:09.080535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.080513 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6" pod "3972e76b-9f4e-4039-a847-9caf4eb841fe" (UID: "3972e76b-9f4e-4039-a847-9caf4eb841fe"). InnerVolumeSpecName "kube-api-access-hx9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:09.179337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.179306 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9g6\" (UniqueName: \"kubernetes.io/projected/3972e76b-9f4e-4039-a847-9caf4eb841fe-kube-api-access-hx9g6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:09.179337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.179334 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:09.179469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.179347 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3972e76b-9f4e-4039-a847-9caf4eb841fe-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:09.783250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.783215 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" event={"ID":"3972e76b-9f4e-4039-a847-9caf4eb841fe","Type":"ContainerDied","Data":"5e9d4163e7bc0d53c5e58a7076314306e4676cd8ab74c9298eb3b2266214bbf3"} Oct 31 19:02:09.783402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.783257 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt" Oct 31 19:02:09.783402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:09.783270 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9d4163e7bc0d53c5e58a7076314306e4676cd8ab74c9298eb3b2266214bbf3" Oct 31 19:02:10.367136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.367100 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg"] Oct 31 19:02:10.367729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.367709 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3972e76b-9f4e-4039-a847-9caf4eb841fe" containerName="revoke-keys" Oct 31 19:02:10.367781 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.367734 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3972e76b-9f4e-4039-a847-9caf4eb841fe" containerName="revoke-keys" Oct 31 19:02:10.367894 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.367882 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3972e76b-9f4e-4039-a847-9caf4eb841fe" containerName="revoke-keys" Oct 31 19:02:10.371623 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.371606 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490192 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490152 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p6x\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490348 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490276 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490348 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490339 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490362 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490453 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490432 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490452 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490477 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490602 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.490660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.490642 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.535467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.535443 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4c58c5-openshift-default-dockercfg-54lm7\"" Oct 31 19:02:10.579077 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.579051 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg"] Oct 31 19:02:10.591524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591497 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p6x\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591641 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591607 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591699 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591658 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591768 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591830 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591858 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591899 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.591956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591933 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.592067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.591961 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.592067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.592052 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.592583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.592521 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.592583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.592529 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.592760 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.592586 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.593170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.593068 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.594555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.594530 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.594777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.594756 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.647451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.647393 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p6x\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.647538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.647513 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:10.684486 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:10.684464 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:11.112288 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:11.112254 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg"] Oct 31 19:02:11.154562 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:02:11.154534 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce634413_5f51_4efb_9370_f9812f2473ba.slice/crio-2f0d3173e8e80db8ff5404c181471a282ab21381c44fa30e3f37825696525416 WatchSource:0}: Error finding container 2f0d3173e8e80db8ff5404c181471a282ab21381c44fa30e3f37825696525416: Status 404 returned error can't find the container with id 2f0d3173e8e80db8ff5404c181471a282ab21381c44fa30e3f37825696525416 Oct 31 19:02:11.156708 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:11.156674 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:11.156805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:11.156748 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:11.156805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:11.156777 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:11.796448 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:11.796416 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" event={"ID":"ce634413-5f51-4efb-9370-f9812f2473ba","Type":"ContainerStarted","Data":"67bd70806d3d356848fa48c66fd9e2a56497d9fe3b4a27562e83a26eff0a594f"} Oct 31 19:02:11.796845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:11.796468 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" event={"ID":"ce634413-5f51-4efb-9370-f9812f2473ba","Type":"ContainerStarted","Data":"2f0d3173e8e80db8ff5404c181471a282ab21381c44fa30e3f37825696525416"} Oct 31 19:02:12.011490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.011443 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podStartSLOduration=2.011431036 podStartE2EDuration="2.011431036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:11.960059081 +0000 UTC m=+2452.562200224" watchObservedRunningTime="2026-10-31 19:02:12.011431036 +0000 UTC m=+2452.613572179" Oct 31 19:02:12.027778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.027752 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw"] Oct 31 19:02:12.031491 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.031477 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.083280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.083221 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a8587d-openshift-default-dockercfg-zdp6b\"" Oct 31 19:02:12.092828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.092804 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw"] Oct 31 19:02:12.208102 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208073 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208117 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208135 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208266 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208288 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208312 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208510 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208407 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208510 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208471 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.208510 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.208489 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkl8h\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.309557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309516 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.309759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309564 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.309759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309629 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.309759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309650 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.309759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309685 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.309759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309728 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309804 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309834 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkl8h\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309905 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309913 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.309976 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.310022 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.310132 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.310576 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.310554 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.312357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.312331 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.312514 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.312498 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.366652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.366579 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.366652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.366628 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkl8h\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.645257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.644671 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:12.684942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.684911 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:12.686614 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.686592 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 19:02:12.686725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:12.686650 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 19:02:13.078502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.078478 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw"] Oct 31 19:02:13.080719 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:02:13.080694 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298fb7e0_12ea_4e91_8142_cb1c27880a69.slice/crio-b43c74578275f23259c90a911cc63b7bcbe5a4292d8b9f64e850ee290f4d0556 WatchSource:0}: Error finding container b43c74578275f23259c90a911cc63b7bcbe5a4292d8b9f64e850ee290f4d0556: Status 404 returned error can't find the container with id b43c74578275f23259c90a911cc63b7bcbe5a4292d8b9f64e850ee290f4d0556 Oct 31 19:02:13.083258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.083220 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:13.083322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.083288 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:13.083322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.083319 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:13.685147 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.685107 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 19:02:13.685362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.685222 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 19:02:13.809284 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.809244 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" event={"ID":"298fb7e0-12ea-4e91-8142-cb1c27880a69","Type":"ContainerStarted","Data":"64cfce986240a0979618a39e0e765bb0fd0f95c5a2bd52d6ebff372a50d69e61"} Oct 31 19:02:13.809463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.809309 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" event={"ID":"298fb7e0-12ea-4e91-8142-cb1c27880a69","Type":"ContainerStarted","Data":"b43c74578275f23259c90a911cc63b7bcbe5a4292d8b9f64e850ee290f4d0556"} Oct 31 19:02:13.930442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:13.930378 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podStartSLOduration=2.930359932 podStartE2EDuration="2.930359932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:13.928089993 +0000 UTC m=+2454.530231137" watchObservedRunningTime="2026-10-31 19:02:13.930359932 +0000 UTC m=+2454.532501077" Oct 31 19:02:14.650218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:14.650163 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:14.650902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:14.650879 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 19:02:14.651047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:14.651025 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 19:02:14.685444 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:14.685406 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 19:02:14.685726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:14.685694 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 19:02:15.652209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:15.645552 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 19:02:15.652209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:15.645608 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 19:02:15.685334 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:15.685304 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 19:02:15.685506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:15.685351 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 19:02:16.645602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:16.645564 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 19:02:16.645908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:16.645879 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 19:02:16.685442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:16.685414 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 19:02:16.685870 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:16.685849 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 19:02:17.653211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:17.650559 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 19:02:17.653211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:17.650621 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 19:02:17.685655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:17.685622 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 19:02:17.686046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:17.685683 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 19:02:18.650573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:18.650536 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 19:02:18.650859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:18.650831 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 19:02:18.688799 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:18.688772 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:18.689174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:18.689153 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:18.689847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:18.689829 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:02:19.645886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:19.645852 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 19:02:19.646052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:19.645907 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 19:02:21.686353 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:21.686314 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:02:21.686766 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:21.686392 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:02:21.741951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:21.741918 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:21.742852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:21.742830 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:21.743880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:21.743858 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:02:24.904021 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:02:24.903987 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:02:26.884708 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:26.884678 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx"] Oct 31 19:02:26.889451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:26.889434 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:26.992020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:26.991995 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:02:26.992020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:26.992016 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:02:26.994340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:26.994324 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx"] Oct 31 19:02:27.073100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.073075 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsnc\" (UniqueName: \"kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.073245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.073120 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.073348 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.073327 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.174399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.174340 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.174502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.174446 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.174502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.174483 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsnc\" (UniqueName: \"kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.175098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.175031 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.176775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.176755 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.230427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.230407 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsnc\" (UniqueName: \"kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc\") pod \"maas-api-revoke-sub-255eadbe-89rjx\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.502437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.502413 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:27.877508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.877484 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx"] Oct 31 19:02:27.879614 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:02:27.879584 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c71a842_a502_4469_bce0_6fabf68dcb65.slice/crio-ab56fedb69fccdfc64e5a1f868bfa52115fd0ecfa6be2ae9cddc79a01ec18631 WatchSource:0}: Error finding container ab56fedb69fccdfc64e5a1f868bfa52115fd0ecfa6be2ae9cddc79a01ec18631: Status 404 returned error can't find the container with id ab56fedb69fccdfc64e5a1f868bfa52115fd0ecfa6be2ae9cddc79a01ec18631 Oct 31 19:02:27.893264 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:27.893229 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" event={"ID":"4c71a842-a502-4469-bce0-6fabf68dcb65","Type":"ContainerStarted","Data":"ab56fedb69fccdfc64e5a1f868bfa52115fd0ecfa6be2ae9cddc79a01ec18631"} Oct 31 19:02:28.901170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:28.901093 5688 generic.go:356] "Generic (PLEG): container finished" podID="4c71a842-a502-4469-bce0-6fabf68dcb65" containerID="b0cf7abe75644b2b5c325226ac88f4761036f01186e8afc368d2e03b3e342ac6" exitCode=0 Oct 31 19:02:28.901608 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:28.901176 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" event={"ID":"4c71a842-a502-4469-bce0-6fabf68dcb65","Type":"ContainerDied","Data":"b0cf7abe75644b2b5c325226ac88f4761036f01186e8afc368d2e03b3e342ac6"} Oct 31 19:02:30.047314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.047293 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:30.107255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.107225 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca\") pod \"4c71a842-a502-4469-bce0-6fabf68dcb65\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " Oct 31 19:02:30.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.107264 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca\") pod \"4c71a842-a502-4469-bce0-6fabf68dcb65\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " Oct 31 19:02:30.107384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.107306 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc\" (UniqueName: \"kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc\") pod \"4c71a842-a502-4469-bce0-6fabf68dcb65\" (UID: \"4c71a842-a502-4469-bce0-6fabf68dcb65\") " Oct 31 19:02:30.107651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.107629 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca" pod "4c71a842-a502-4469-bce0-6fabf68dcb65" (UID: "4c71a842-a502-4469-bce0-6fabf68dcb65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:02:30.109469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.109444 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca" pod "4c71a842-a502-4469-bce0-6fabf68dcb65" (UID: "4c71a842-a502-4469-bce0-6fabf68dcb65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:02:30.109627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.109604 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc" pod "4c71a842-a502-4469-bce0-6fabf68dcb65" (UID: "4c71a842-a502-4469-bce0-6fabf68dcb65"). InnerVolumeSpecName "kube-api-access-5nsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:30.208711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.208640 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:30.208711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.208676 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c71a842-a502-4469-bce0-6fabf68dcb65-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:30.208711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.208687 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nsnc\" (UniqueName: \"kubernetes.io/projected/4c71a842-a502-4469-bce0-6fabf68dcb65-kube-api-access-5nsnc\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:30.919032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.919005 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" Oct 31 19:02:30.919392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.919046 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx" event={"ID":"4c71a842-a502-4469-bce0-6fabf68dcb65","Type":"ContainerDied","Data":"ab56fedb69fccdfc64e5a1f868bfa52115fd0ecfa6be2ae9cddc79a01ec18631"} Oct 31 19:02:30.919392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:30.919377 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab56fedb69fccdfc64e5a1f868bfa52115fd0ecfa6be2ae9cddc79a01ec18631" Oct 31 19:02:40.366742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.366711 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d"] Oct 31 19:02:40.367229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.367209 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4c71a842-a502-4469-bce0-6fabf68dcb65" containerName="revoke-keys" Oct 31 19:02:40.367229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.367222 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c71a842-a502-4469-bce0-6fabf68dcb65" containerName="revoke-keys" Oct 31 19:02:40.367322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.367312 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c71a842-a502-4469-bce0-6fabf68dcb65" containerName="revoke-keys" Oct 31 19:02:40.370724 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.370706 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419209 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419247 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419276 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbtq\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419363 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419378 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419402 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419420 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419525 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.419582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.419552 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520302 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520271 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520311 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520344 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520448 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520556 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520588 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520624 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbtq\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520678 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520702 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520754 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.520846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520778 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.521016 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.520990 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.521157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.521029 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.521431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.521408 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.522755 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.522731 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.523133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.523115 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.574289 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.574260 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d"] Oct 31 19:02:40.577498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.577468 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.577640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.577617 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbtq\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:40.683437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:40.683356 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:41.156060 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.154554 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d"] Oct 31 19:02:41.159083 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.159043 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:41.159211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.159135 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:41.159211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.159200 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:41.470648 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.470565 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk"] Oct 31 19:02:41.474645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.474621 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.541214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.529372 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:02:41.541214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.530363 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b68\" (UniqueName: \"kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.541214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.530438 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.541214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.530650 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.542072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.542041 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk"] Oct 31 19:02:41.569374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.569343 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:02:41.632046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.632015 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.640775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.632078 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b68\" (UniqueName: \"kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.640775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.632111 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.642881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.642852 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.658626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.658597 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.688302 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.688274 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b68\" (UniqueName: \"kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68\") pod \"maas-api-revoke-sub-be0e0a99-r7lnk\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.787705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.787664 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:41.994932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.994900 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" event={"ID":"0b137635-a697-45c9-8196-cd213f057873","Type":"ContainerStarted","Data":"e42e17b4e94048608a778fafdb77b6aace7a1fa8d11e1152430c77ce78d44eab"} Oct 31 19:02:41.995145 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:41.994949 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" event={"ID":"0b137635-a697-45c9-8196-cd213f057873","Type":"ContainerStarted","Data":"dc7efa1e350363dcf20ea8876990b6a5ac12fbdd96cb931397f362a4ba231db9"} Oct 31 19:02:42.155269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:42.153014 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podStartSLOduration=2.152993477 podStartE2EDuration="2.152993477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:42.150656148 +0000 UTC m=+2482.752797295" watchObservedRunningTime="2026-10-31 19:02:42.152993477 +0000 UTC m=+2482.755134622" Oct 31 19:02:42.205793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:42.205488 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk"] Oct 31 19:02:42.683640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:42.683601 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:42.685426 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:42.685394 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:02:42.685544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:42.685449 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:02:43.002227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:43.002177 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" event={"ID":"9d1deb11-7cd9-4fa0-9305-5d3abfd05692","Type":"ContainerStarted","Data":"4569794f891ea51cce4ddc98dbac70144909c8721af6ee99e67c512f60c8639e"} Oct 31 19:02:43.002387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:43.002253 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" event={"ID":"9d1deb11-7cd9-4fa0-9305-5d3abfd05692","Type":"ContainerStarted","Data":"7543b14ab8648518e388bdb6e045bd9924082b3b2b5ae5f544439d3727b9bb73"} Oct 31 19:02:43.119603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:43.119554 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" podStartSLOduration=2.119539188 podStartE2EDuration="2.119539188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:43.117583199 +0000 UTC m=+2483.719724342" watchObservedRunningTime="2026-10-31 19:02:43.119539188 +0000 UTC m=+2483.721680350" Oct 31 19:02:43.684537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:43.684446 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:02:43.684537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:43.684514 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:02:44.009674 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:44.009640 5688 generic.go:356] "Generic (PLEG): container finished" podID="9d1deb11-7cd9-4fa0-9305-5d3abfd05692" containerID="4569794f891ea51cce4ddc98dbac70144909c8721af6ee99e67c512f60c8639e" exitCode=0 Oct 31 19:02:44.009895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:44.009719 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" event={"ID":"9d1deb11-7cd9-4fa0-9305-5d3abfd05692","Type":"ContainerDied","Data":"4569794f891ea51cce4ddc98dbac70144909c8721af6ee99e67c512f60c8639e"} Oct 31 19:02:44.684347 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:44.684320 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:02:44.684551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:44.684368 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:02:45.222134 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.222104 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:45.317935 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.317862 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca\") pod \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " Oct 31 19:02:45.318084 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.317937 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68\" (UniqueName: \"kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68\") pod \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " Oct 31 19:02:45.318084 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.317964 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca\") pod \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\" (UID: \"9d1deb11-7cd9-4fa0-9305-5d3abfd05692\") " Oct 31 19:02:45.318427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.318395 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca" pod "9d1deb11-7cd9-4fa0-9305-5d3abfd05692" (UID: "9d1deb11-7cd9-4fa0-9305-5d3abfd05692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:02:45.320154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.320128 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca" pod "9d1deb11-7cd9-4fa0-9305-5d3abfd05692" (UID: "9d1deb11-7cd9-4fa0-9305-5d3abfd05692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:02:45.320254 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.320171 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68" pod "9d1deb11-7cd9-4fa0-9305-5d3abfd05692" (UID: "9d1deb11-7cd9-4fa0-9305-5d3abfd05692"). InnerVolumeSpecName "kube-api-access-z6b68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:45.419719 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.419684 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:45.419719 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.419714 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b68\" (UniqueName: \"kubernetes.io/projected/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-kube-api-access-z6b68\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:45.419928 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.419730 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1deb11-7cd9-4fa0-9305-5d3abfd05692-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:45.684657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.684627 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:02:45.685063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:45.684678 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:02:46.022098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:46.022064 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" event={"ID":"9d1deb11-7cd9-4fa0-9305-5d3abfd05692","Type":"ContainerDied","Data":"7543b14ab8648518e388bdb6e045bd9924082b3b2b5ae5f544439d3727b9bb73"} Oct 31 19:02:46.022098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:46.022089 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk" Oct 31 19:02:46.022331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:46.022115 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7543b14ab8648518e388bdb6e045bd9924082b3b2b5ae5f544439d3727b9bb73" Oct 31 19:02:46.684304 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:46.684273 5688 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:02:46.684471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:46.684322 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:02:47.477787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:47.477710 5688 patch_prober.go:29] interesting pod/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 19:02:47.477787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:47.477766 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 19:02:47.688141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:47.688065 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:47.688464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:47.688446 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:47.689112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:47.689096 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:02:48.035884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:48.035853 5688 generic.go:356] "Generic (PLEG): container finished" podID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerID="2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3" exitCode=0 Oct 31 19:02:48.036080 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:48.035927 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" event={"ID":"169b7f77-ea2e-4d64-925c-eb082102aa78","Type":"ContainerDied","Data":"2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3"} Oct 31 19:02:48.036489 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:48.036473 5688 scope.go:122] "RemoveContainer" containerID="2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3" Oct 31 19:02:48.037912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:48.037875 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:48.037999 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:48.037969 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:48.038039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:48.038016 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:49.065089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:49.065052 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb"] Oct 31 19:02:49.067905 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:49.067873 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" event={"ID":"169b7f77-ea2e-4d64-925c-eb082102aa78","Type":"ContainerStarted","Data":"53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159"} Oct 31 19:02:49.166360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:49.166318 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj"] Oct 31 19:02:49.763433 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:49.763390 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:02:49.763640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:49.763536 5688 patch_prober.go:29] interesting pod/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 19:02:49.763640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:49.763618 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 19:02:50.763665 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:50.763616 5688 patch_prober.go:29] interesting pod/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 19:02:50.764349 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:50.764314 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 19:02:51.761938 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:51.761902 5688 patch_prober.go:29] interesting pod/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 19:02:51.762116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:51.761955 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 19:02:52.761793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:52.761763 5688 patch_prober.go:29] interesting pod/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 19:02:52.762219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:52.761817 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 19:02:53.763904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:53.763870 5688 patch_prober.go:29] interesting pod/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 19:02:53.764390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:53.763944 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 19:02:53.832216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:53.832169 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59"] Oct 31 19:02:53.832412 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:53.832354 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" containerID="cri-o://4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a" gracePeriod=30 Oct 31 19:02:54.065762 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.065652 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" podUID="273a43c7-34c0-4fea-8684-f598407e8aba" containerName="payload-processing" containerID="cri-o://8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d" gracePeriod=25 Oct 31 19:02:54.166851 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.166804 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" podUID="e4938e46-aa53-49cd-bc2c-ba916d4e6077" containerName="payload-pre-processing" containerID="cri-o://8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4" gracePeriod=25 Oct 31 19:02:54.329901 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.329845 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:02:54.423730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.423706 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:02:54.485819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.485790 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume\") pod \"273a43c7-34c0-4fea-8684-f598407e8aba\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " Oct 31 19:02:54.485943 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.485914 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh\" (UniqueName: \"kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh\") pod \"273a43c7-34c0-4fea-8684-f598407e8aba\" (UID: \"273a43c7-34c0-4fea-8684-f598407e8aba\") " Oct 31 19:02:54.486133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.486112 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume" pod "273a43c7-34c0-4fea-8684-f598407e8aba" (UID: "273a43c7-34c0-4fea-8684-f598407e8aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:02:54.486250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.486236 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273a43c7-34c0-4fea-8684-f598407e8aba-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:54.487998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.487980 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh" pod "273a43c7-34c0-4fea-8684-f598407e8aba" (UID: "273a43c7-34c0-4fea-8684-f598407e8aba"). InnerVolumeSpecName "kube-api-access-rjzlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:54.587427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.587358 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz\" (UniqueName: \"kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz\") pod \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " Oct 31 19:02:54.587554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.587526 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume\") pod \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\" (UID: \"e4938e46-aa53-49cd-bc2c-ba916d4e6077\") " Oct 31 19:02:54.587872 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.587850 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume" pod "e4938e46-aa53-49cd-bc2c-ba916d4e6077" (UID: "e4938e46-aa53-49cd-bc2c-ba916d4e6077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:02:54.587982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.587878 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjzlh\" (UniqueName: \"kubernetes.io/projected/273a43c7-34c0-4fea-8684-f598407e8aba-kube-api-access-rjzlh\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:54.589676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.589645 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz" pod "e4938e46-aa53-49cd-bc2c-ba916d4e6077" (UID: "e4938e46-aa53-49cd-bc2c-ba916d4e6077"). InnerVolumeSpecName "kube-api-access-h8wtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:54.689085 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.689059 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4938e46-aa53-49cd-bc2c-ba916d4e6077-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:54.689085 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.689081 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wtz\" (UniqueName: \"kubernetes.io/projected/e4938e46-aa53-49cd-bc2c-ba916d4e6077-kube-api-access-h8wtz\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:54.766463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.766440 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:02:54.766825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.766719 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:02:54.767458 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:54.767442 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:02:54.903413 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:02:54.903344 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:02:55.109005 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.108975 5688 generic.go:356] "Generic (PLEG): container finished" podID="273a43c7-34c0-4fea-8684-f598407e8aba" containerID="8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d" exitCode=0 Oct 31 19:02:55.109223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.109030 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" Oct 31 19:02:55.109223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.109051 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" event={"ID":"273a43c7-34c0-4fea-8684-f598407e8aba","Type":"ContainerDied","Data":"8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d"} Oct 31 19:02:55.109223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.109102 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb" event={"ID":"273a43c7-34c0-4fea-8684-f598407e8aba","Type":"ContainerDied","Data":"5e5904de13847523bf9ee7283939d6a3625306cb82b1895fc508d9e8aba4f31f"} Oct 31 19:02:55.109223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.109129 5688 scope.go:122] "RemoveContainer" containerID="8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d" Oct 31 19:02:55.110603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.110402 5688 generic.go:356] "Generic (PLEG): container finished" podID="e4938e46-aa53-49cd-bc2c-ba916d4e6077" containerID="8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4" exitCode=0 Oct 31 19:02:55.110603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.110444 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" Oct 31 19:02:55.110603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.110465 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" event={"ID":"e4938e46-aa53-49cd-bc2c-ba916d4e6077","Type":"ContainerDied","Data":"8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4"} Oct 31 19:02:55.110603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.110508 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj" event={"ID":"e4938e46-aa53-49cd-bc2c-ba916d4e6077","Type":"ContainerDied","Data":"a414357460948d2de698a859ebdf1efb84479be3ddc21ed775a54fd03d923148"} Oct 31 19:02:55.119988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.119967 5688 scope.go:122] "RemoveContainer" containerID="8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d" Oct 31 19:02:55.120346 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:02:55.120321 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d\": container with ID starting with 8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d not found: ID does not exist" containerID="8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d" Oct 31 19:02:55.120445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.120360 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d"} err="failed to get container status \"8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d\": rpc error: code = NotFound desc = could not find container \"8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d\": container with ID starting with 8f9e646e760f3976b15cc770432f7a645d5e059af69c6fa73addf7236d40824d not found: ID does not exist" Oct 31 19:02:55.120445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.120383 5688 scope.go:122] "RemoveContainer" containerID="8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4" Oct 31 19:02:55.128694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.128678 5688 scope.go:122] "RemoveContainer" containerID="8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4" Oct 31 19:02:55.128944 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:02:55.128928 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4\": container with ID starting with 8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4 not found: ID does not exist" containerID="8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4" Oct 31 19:02:55.129017 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.128951 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4"} err="failed to get container status \"8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4\": rpc error: code = NotFound desc = could not find container \"8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4\": container with ID starting with 8bba63b47031f5b1a75f73f7249e225ef014c33ecf86c5e130068aa51c3b71c4 not found: ID does not exist" Oct 31 19:02:55.271146 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.271118 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj"] Oct 31 19:02:55.275451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.275424 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4c58c5-597b78cd79-gz4zj"] Oct 31 19:02:55.383223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.383165 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb"] Oct 31 19:02:55.385263 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.385243 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4c58c5-7c74f99f55-62wnb"] Oct 31 19:02:55.565299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.565221 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273a43c7-34c0-4fea-8684-f598407e8aba" path="/var/lib/kubelet/pods/273a43c7-34c0-4fea-8684-f598407e8aba/volumes" Oct 31 19:02:55.565689 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:55.565666 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4938e46-aa53-49cd-bc2c-ba916d4e6077" path="/var/lib/kubelet/pods/e4938e46-aa53-49cd-bc2c-ba916d4e6077/volumes" Oct 31 19:02:57.371520 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.371483 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g"] Oct 31 19:02:57.371955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.371941 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e4938e46-aa53-49cd-bc2c-ba916d4e6077" containerName="payload-pre-processing" Oct 31 19:02:57.372001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.371957 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4938e46-aa53-49cd-bc2c-ba916d4e6077" containerName="payload-pre-processing" Oct 31 19:02:57.372035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372007 5688 cpu_manager.go:414] "containerMap: removing container" podUID="273a43c7-34c0-4fea-8684-f598407e8aba" containerName="payload-processing" Oct 31 19:02:57.372035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372012 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273a43c7-34c0-4fea-8684-f598407e8aba" containerName="payload-processing" Oct 31 19:02:57.372035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372026 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9d1deb11-7cd9-4fa0-9305-5d3abfd05692" containerName="revoke-keys" Oct 31 19:02:57.372035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372031 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1deb11-7cd9-4fa0-9305-5d3abfd05692" containerName="revoke-keys" Oct 31 19:02:57.372222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372116 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1deb11-7cd9-4fa0-9305-5d3abfd05692" containerName="revoke-keys" Oct 31 19:02:57.372222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372129 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4938e46-aa53-49cd-bc2c-ba916d4e6077" containerName="payload-pre-processing" Oct 31 19:02:57.372222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.372139 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="273a43c7-34c0-4fea-8684-f598407e8aba" containerName="payload-processing" Oct 31 19:02:57.374965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.374947 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519263 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519231 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519269 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519361 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519392 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q955s\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519414 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519506 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519531 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519586 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.519697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.519667 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.580220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.580164 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g"] Oct 31 19:02:57.620705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620676 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.620847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620710 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q955s\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.620847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620732 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.620847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620763 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.620847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620783 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620863 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620889 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.620971 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.621006 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621300 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.621071 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621300 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.621174 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621419 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.621324 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.621550 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.621892 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.621870 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.623228 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.623177 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.623420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.623403 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.676955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.676929 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.677076 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.676932 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q955s\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:57.703653 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:57.703348 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:58.134583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:58.134560 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g"] Oct 31 19:02:58.136897 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:02:58.136869 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f697de_be92_4ec9_bd0b_05fce7f411f8.slice/crio-732a9b2a86d12280adea2e0f1d6a75f28661d6068986a6221b0d8c80f8dbf433 WatchSource:0}: Error finding container 732a9b2a86d12280adea2e0f1d6a75f28661d6068986a6221b0d8c80f8dbf433: Status 404 returned error can't find the container with id 732a9b2a86d12280adea2e0f1d6a75f28661d6068986a6221b0d8c80f8dbf433 Oct 31 19:02:58.139108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:58.139078 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:58.139225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:58.139140 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:58.139225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:58.139169 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:02:59.127010 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.126977 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 19:02:59.163138 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.163092 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerID="4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a" exitCode=0 Oct 31 19:02:59.163317 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.163233 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" Oct 31 19:02:59.163317 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.163230 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" event={"ID":"1f9edaaa-f008-458b-9b08-c8bd2077d6bc","Type":"ContainerDied","Data":"4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a"} Oct 31 19:02:59.163434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.163318 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59" event={"ID":"1f9edaaa-f008-458b-9b08-c8bd2077d6bc","Type":"ContainerDied","Data":"4ec820212238cec9eded49cf52342953989f7220f1f7f7a12cbeca1c049e6a17"} Oct 31 19:02:59.163434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.163348 5688 scope.go:122] "RemoveContainer" containerID="4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a" Oct 31 19:02:59.164899 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.164881 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" event={"ID":"c9f697de-be92-4ec9-bd0b-05fce7f411f8","Type":"ContainerStarted","Data":"45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee"} Oct 31 19:02:59.164993 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.164915 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" event={"ID":"c9f697de-be92-4ec9-bd0b-05fce7f411f8","Type":"ContainerStarted","Data":"732a9b2a86d12280adea2e0f1d6a75f28661d6068986a6221b0d8c80f8dbf433"} Oct 31 19:02:59.174078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.174061 5688 scope.go:122] "RemoveContainer" containerID="ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8" Oct 31 19:02:59.183010 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.182996 5688 scope.go:122] "RemoveContainer" containerID="4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a" Oct 31 19:02:59.183295 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:02:59.183271 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a\": container with ID starting with 4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a not found: ID does not exist" containerID="4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a" Oct 31 19:02:59.183384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.183309 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a"} err="failed to get container status \"4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a\": rpc error: code = NotFound desc = could not find container \"4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a\": container with ID starting with 4783c29ea9148445186906dfd7e08639521b79349fd5ae77964ec83ccb741f8a not found: ID does not exist" Oct 31 19:02:59.183384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.183331 5688 scope.go:122] "RemoveContainer" containerID="ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8" Oct 31 19:02:59.183589 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:02:59.183572 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8\": container with ID starting with ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8 not found: ID does not exist" containerID="ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8" Oct 31 19:02:59.183632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.183595 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8"} err="failed to get container status \"ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8\": rpc error: code = NotFound desc = could not find container \"ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8\": container with ID starting with ba2ea414897e365822e77a7f0579278956c3a505a282373648bb707d016b79d8 not found: ID does not exist" Oct 31 19:02:59.258990 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.258958 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.258996 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259084 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259112 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259224 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259257 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259296 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259365 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259403 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket\") pod \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\" (UID: \"1f9edaaa-f008-458b-9b08-c8bd2077d6bc\") " Oct 31 19:02:59.259657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259598 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:02:59.259785 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.259762 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:02:59.260206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.260161 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:02:59.260359 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.260334 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:02:59.260524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.260484 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.260597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.260526 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.260597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.260544 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.260597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.260500 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:02:59.263311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.262729 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "kube-api-access-zt766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:59.263311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.262780 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:02:59.263311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.263263 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:02:59.263311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.263302 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy" pod "1f9edaaa-f008-458b-9b08-c8bd2077d6bc" (UID: "1f9edaaa-f008-458b-9b08-c8bd2077d6bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:02:59.361615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.361530 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.361615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.361565 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.361615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.361583 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt766\" (UniqueName: \"kubernetes.io/projected/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-kube-api-access-zt766\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.361615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.361598 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.361615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.361612 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.361922 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.361629 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9edaaa-f008-458b-9b08-c8bd2077d6bc-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:02:59.413753 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.413707 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podStartSLOduration=2.413695612 podStartE2EDuration="2.413695612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:59.412683457 +0000 UTC m=+2500.014824593" watchObservedRunningTime="2026-10-31 19:02:59.413695612 +0000 UTC m=+2500.015836749" Oct 31 19:02:59.582150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.582120 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59"] Oct 31 19:02:59.589523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.589488 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e43ccf-openshift-default-6b958ffdc7-22f59"] Oct 31 19:02:59.704338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.704251 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:02:59.705717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.705690 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:02:59.705860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:02:59.705754 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:03:00.704305 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:00.704272 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:03:00.704693 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:00.704323 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:03:01.566802 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:01.566766 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" path="/var/lib/kubelet/pods/1f9edaaa-f008-458b-9b08-c8bd2077d6bc/volumes" Oct 31 19:03:01.704840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:01.704811 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:03:01.705266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:01.704865 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:03:02.704834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:02.704796 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:03:02.705027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:02.704860 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:03:03.704502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:03.704470 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:03:03.704714 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:03.704527 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:03:04.704070 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:04.704038 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:03:04.704506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:04.704106 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:03:05.708336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:05.708305 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:03:05.708803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:05.708702 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:03:05.709389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:05.709370 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:03:07.465247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.461242 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz"] Oct 31 19:03:07.465247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.462627 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" podUID="81e38397-012b-4b6c-b2a7-8e0681d7c894" containerName="payload-processing" containerID="cri-o://32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8" gracePeriod=30 Oct 31 19:03:07.777270 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.777242 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:03:07.884046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.884017 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq\" (UniqueName: \"kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq\") pod \"81e38397-012b-4b6c-b2a7-8e0681d7c894\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " Oct 31 19:03:07.884239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.884158 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume\") pod \"81e38397-012b-4b6c-b2a7-8e0681d7c894\" (UID: \"81e38397-012b-4b6c-b2a7-8e0681d7c894\") " Oct 31 19:03:07.884546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.884515 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume" pod "81e38397-012b-4b6c-b2a7-8e0681d7c894" (UID: "81e38397-012b-4b6c-b2a7-8e0681d7c894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:07.886209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.886163 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq" pod "81e38397-012b-4b6c-b2a7-8e0681d7c894" (UID: "81e38397-012b-4b6c-b2a7-8e0681d7c894"). InnerVolumeSpecName "kube-api-access-5b2fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:07.985532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.985494 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e38397-012b-4b6c-b2a7-8e0681d7c894-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:07.985532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:07.985524 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b2fq\" (UniqueName: \"kubernetes.io/projected/81e38397-012b-4b6c-b2a7-8e0681d7c894-kube-api-access-5b2fq\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:08.275902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.275867 5688 generic.go:356] "Generic (PLEG): container finished" podID="81e38397-012b-4b6c-b2a7-8e0681d7c894" containerID="32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8" exitCode=0 Oct 31 19:03:08.276111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.275916 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" Oct 31 19:03:08.276111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.275962 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" event={"ID":"81e38397-012b-4b6c-b2a7-8e0681d7c894","Type":"ContainerDied","Data":"32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8"} Oct 31 19:03:08.276111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.276017 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz" event={"ID":"81e38397-012b-4b6c-b2a7-8e0681d7c894","Type":"ContainerDied","Data":"2afe28869bf0dfdd65489966bfeed5106311856c1d4835d021256cdfb505b68c"} Oct 31 19:03:08.276111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.276045 5688 scope.go:122] "RemoveContainer" containerID="32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8" Oct 31 19:03:08.286123 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.286105 5688 scope.go:122] "RemoveContainer" containerID="32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8" Oct 31 19:03:08.286391 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:03:08.286373 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8\": container with ID starting with 32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8 not found: ID does not exist" containerID="32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8" Oct 31 19:03:08.286442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.286400 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8"} err="failed to get container status \"32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8\": rpc error: code = NotFound desc = could not find container \"32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8\": container with ID starting with 32c0ac13c0ebb314542924f8657f9faf7485475b39031fb2993ba6de48c2ead8 not found: ID does not exist" Oct 31 19:03:08.354078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354052 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps"] Oct 31 19:03:08.354502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354488 5688 cpu_manager.go:414] "containerMap: removing container" podUID="81e38397-012b-4b6c-b2a7-8e0681d7c894" containerName="payload-processing" Oct 31 19:03:08.354551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354503 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e38397-012b-4b6c-b2a7-8e0681d7c894" containerName="payload-processing" Oct 31 19:03:08.354551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354514 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" Oct 31 19:03:08.354551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354521 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" Oct 31 19:03:08.354551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354541 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" Oct 31 19:03:08.354551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354546 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" Oct 31 19:03:08.354695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354622 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" Oct 31 19:03:08.354695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354630 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e38397-012b-4b6c-b2a7-8e0681d7c894" containerName="payload-processing" Oct 31 19:03:08.354695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.354638 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9edaaa-f008-458b-9b08-c8bd2077d6bc" containerName="istio-proxy" Oct 31 19:03:08.359024 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.359005 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.396574 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.396544 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.396751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.396594 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.396751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.396701 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24q8\" (UniqueName: \"kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.410964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.410940 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:03:08.411172 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.411158 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:03:08.449032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.448998 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps"] Oct 31 19:03:08.452145 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.452117 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz"] Oct 31 19:03:08.458486 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.458458 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-6cbc876868-zjdfz"] Oct 31 19:03:08.498018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.497987 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.498524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.498063 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.498524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.498112 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24q8\" (UniqueName: \"kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.498849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.498825 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.500463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.500435 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.555363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.555296 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24q8\" (UniqueName: \"kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8\") pod \"maas-api-revoke-sub-dbd09fa9-mp8ps\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:08.672604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:08.672567 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:09.050068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:09.050043 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps"] Oct 31 19:03:09.053061 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:09.053029 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ba62d2_b6e6_430d_b52a_2ee9b9d1d0cf.slice/crio-f41c950029567c427c700a775e767f3c8899b3987058ccd7fb9a02f13f7d8836 WatchSource:0}: Error finding container f41c950029567c427c700a775e767f3c8899b3987058ccd7fb9a02f13f7d8836: Status 404 returned error can't find the container with id f41c950029567c427c700a775e767f3c8899b3987058ccd7fb9a02f13f7d8836 Oct 31 19:03:09.283980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:09.283949 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" event={"ID":"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf","Type":"ContainerStarted","Data":"f41c950029567c427c700a775e767f3c8899b3987058ccd7fb9a02f13f7d8836"} Oct 31 19:03:09.565797 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:09.565765 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e38397-012b-4b6c-b2a7-8e0681d7c894" path="/var/lib/kubelet/pods/81e38397-012b-4b6c-b2a7-8e0681d7c894/volumes" Oct 31 19:03:10.293284 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:10.293242 5688 generic.go:356] "Generic (PLEG): container finished" podID="18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" containerID="286fdd4e3567275784323a6dbdc7ac4e60c58288a7f7e4dcf623153d78d75569" exitCode=0 Oct 31 19:03:10.293456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:10.293331 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" event={"ID":"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf","Type":"ContainerDied","Data":"286fdd4e3567275784323a6dbdc7ac4e60c58288a7f7e4dcf623153d78d75569"} Oct 31 19:03:11.450727 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.450703 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:11.534739 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.534708 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca\") pod \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " Oct 31 19:03:11.534915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.534864 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8\" (UniqueName: \"kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8\") pod \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " Oct 31 19:03:11.534915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.534891 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca\") pod \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\" (UID: \"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf\") " Oct 31 19:03:11.535371 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.535333 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca" pod "18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" (UID: "18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:11.537006 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.536985 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca" pod "18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" (UID: "18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:03:11.537065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.537006 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8" pod "18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" (UID: "18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf"). InnerVolumeSpecName "kube-api-access-l24q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:11.636440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.636401 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24q8\" (UniqueName: \"kubernetes.io/projected/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-kube-api-access-l24q8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:11.636440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.636432 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:11.636653 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:11.636448 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:12.311101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:12.311064 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" event={"ID":"18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf","Type":"ContainerDied","Data":"f41c950029567c427c700a775e767f3c8899b3987058ccd7fb9a02f13f7d8836"} Oct 31 19:03:12.311376 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:12.311356 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps" Oct 31 19:03:12.311590 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:12.311429 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f41c950029567c427c700a775e767f3c8899b3987058ccd7fb9a02f13f7d8836" Oct 31 19:03:16.237487 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.237446 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq"] Oct 31 19:03:16.238198 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.238156 5688 cpu_manager.go:414] "containerMap: removing container" podUID="18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" containerName="revoke-keys" Oct 31 19:03:16.238319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.238204 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" containerName="revoke-keys" Oct 31 19:03:16.238416 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.238402 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" containerName="revoke-keys" Oct 31 19:03:16.241099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.241081 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.292944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.292910 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:03:16.342954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.342924 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:03:16.346063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.346032 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq"] Oct 31 19:03:16.400897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.400864 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6wg\" (UniqueName: \"kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.401063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.400971 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.401063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.401025 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.501917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.501839 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6wg\" (UniqueName: \"kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.502065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.501920 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.502065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.501961 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.502581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.502562 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.504401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.504378 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.558644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.558612 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6wg\" (UniqueName: \"kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg\") pod \"maas-api-revoke-sub-816afeb5-b4bjq\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:16.852826 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:16.852756 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:17.231564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:17.231536 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq"] Oct 31 19:03:17.233871 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:17.233832 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53f6490_d12c_40f5_aa31_a4ba5600bc8b.slice/crio-2a8fe728c30bfa21e0fe17579f9fde4537790f1907a36a7cba61f6b0b611d248 WatchSource:0}: Error finding container 2a8fe728c30bfa21e0fe17579f9fde4537790f1907a36a7cba61f6b0b611d248: Status 404 returned error can't find the container with id 2a8fe728c30bfa21e0fe17579f9fde4537790f1907a36a7cba61f6b0b611d248 Oct 31 19:03:17.367460 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:17.367423 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" event={"ID":"e53f6490-d12c-40f5-aa31-a4ba5600bc8b","Type":"ContainerStarted","Data":"2a8fe728c30bfa21e0fe17579f9fde4537790f1907a36a7cba61f6b0b611d248"} Oct 31 19:03:18.374458 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:18.374366 5688 generic.go:356] "Generic (PLEG): container finished" podID="e53f6490-d12c-40f5-aa31-a4ba5600bc8b" containerID="3b580c5d5989320448e545584d325a26154377eaf6e4c3c8963cc27efe0aaaa6" exitCode=0 Oct 31 19:03:18.374458 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:18.374434 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" event={"ID":"e53f6490-d12c-40f5-aa31-a4ba5600bc8b","Type":"ContainerDied","Data":"3b580c5d5989320448e545584d325a26154377eaf6e4c3c8963cc27efe0aaaa6"} Oct 31 19:03:19.549584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.549557 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:19.673256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.656975 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca\") pod \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " Oct 31 19:03:19.673256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.657172 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg\" (UniqueName: \"kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg\") pod \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " Oct 31 19:03:19.673256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.657272 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca\") pod \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\" (UID: \"e53f6490-d12c-40f5-aa31-a4ba5600bc8b\") " Oct 31 19:03:19.709772 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.709705 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca" pod "e53f6490-d12c-40f5-aa31-a4ba5600bc8b" (UID: "e53f6490-d12c-40f5-aa31-a4ba5600bc8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:19.717813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.717768 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca" pod "e53f6490-d12c-40f5-aa31-a4ba5600bc8b" (UID: "e53f6490-d12c-40f5-aa31-a4ba5600bc8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:03:19.722583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.722525 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg" pod "e53f6490-d12c-40f5-aa31-a4ba5600bc8b" (UID: "e53f6490-d12c-40f5-aa31-a4ba5600bc8b"). InnerVolumeSpecName "kube-api-access-vg6wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:19.758841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.758807 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:19.758841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.758838 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6wg\" (UniqueName: \"kubernetes.io/projected/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-kube-api-access-vg6wg\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:19.758841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:19.758848 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53f6490-d12c-40f5-aa31-a4ba5600bc8b-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:20.386742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:20.386704 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" event={"ID":"e53f6490-d12c-40f5-aa31-a4ba5600bc8b","Type":"ContainerDied","Data":"2a8fe728c30bfa21e0fe17579f9fde4537790f1907a36a7cba61f6b0b611d248"} Oct 31 19:03:20.386919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:20.386760 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8fe728c30bfa21e0fe17579f9fde4537790f1907a36a7cba61f6b0b611d248" Oct 31 19:03:20.386919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:20.386715 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq" Oct 31 19:03:24.905148 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:03:24.905108 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:03:27.013347 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.013313 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs"] Oct 31 19:03:27.013755 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.013741 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e53f6490-d12c-40f5-aa31-a4ba5600bc8b" containerName="revoke-keys" Oct 31 19:03:27.013812 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.013757 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53f6490-d12c-40f5-aa31-a4ba5600bc8b" containerName="revoke-keys" Oct 31 19:03:27.013878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.013870 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53f6490-d12c-40f5-aa31-a4ba5600bc8b" containerName="revoke-keys" Oct 31 19:03:27.018356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.018342 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.073819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.073788 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs"] Oct 31 19:03:27.120020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.119985 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:03:27.120020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.120005 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:03:27.152399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.152376 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.152567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.152425 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.152567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.152444 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kxw\" (UniqueName: \"kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.253884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.253853 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.253884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.253886 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kxw\" (UniqueName: \"kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.254110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.253991 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.254567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.254540 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.256461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.256438 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.310597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.310534 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kxw\" (UniqueName: \"kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw\") pod \"maas-api-revoke-sub-c7457c71-62gfs\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.330828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.330805 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:27.701481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:27.701458 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs"] Oct 31 19:03:27.703481 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:27.703454 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e63562_ae59_4132_81d2_4389507b540e.slice/crio-07eaec2b90581c12dab27f90a80abd9c616f71092c815b0274609f49c1be606e WatchSource:0}: Error finding container 07eaec2b90581c12dab27f90a80abd9c616f71092c815b0274609f49c1be606e: Status 404 returned error can't find the container with id 07eaec2b90581c12dab27f90a80abd9c616f71092c815b0274609f49c1be606e Oct 31 19:03:28.435625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:28.435580 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" event={"ID":"81e63562-ae59-4132-81d2-4389507b540e","Type":"ContainerStarted","Data":"44255d57908ca7d3f8c03d52a6acd3e60a2ee3c86d99270210d5aa3d1494b5b7"} Oct 31 19:03:28.436018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:28.435641 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" event={"ID":"81e63562-ae59-4132-81d2-4389507b540e","Type":"ContainerStarted","Data":"07eaec2b90581c12dab27f90a80abd9c616f71092c815b0274609f49c1be606e"} Oct 31 19:03:28.550057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:28.550007 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" podStartSLOduration=2.549993497 podStartE2EDuration="2.549993497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:28.548156966 +0000 UTC m=+2529.150298104" watchObservedRunningTime="2026-10-31 19:03:28.549993497 +0000 UTC m=+2529.152134682" Oct 31 19:03:29.443091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:29.443060 5688 generic.go:356] "Generic (PLEG): container finished" podID="81e63562-ae59-4132-81d2-4389507b540e" containerID="44255d57908ca7d3f8c03d52a6acd3e60a2ee3c86d99270210d5aa3d1494b5b7" exitCode=0 Oct 31 19:03:29.443532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:29.443136 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" event={"ID":"81e63562-ae59-4132-81d2-4389507b540e","Type":"ContainerDied","Data":"44255d57908ca7d3f8c03d52a6acd3e60a2ee3c86d99270210d5aa3d1494b5b7"} Oct 31 19:03:30.668059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.668029 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:30.801609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.801569 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca\") pod \"81e63562-ae59-4132-81d2-4389507b540e\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " Oct 31 19:03:30.801837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.801672 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca\") pod \"81e63562-ae59-4132-81d2-4389507b540e\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " Oct 31 19:03:30.801837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.801744 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw\" (UniqueName: \"kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw\") pod \"81e63562-ae59-4132-81d2-4389507b540e\" (UID: \"81e63562-ae59-4132-81d2-4389507b540e\") " Oct 31 19:03:30.804927 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.804893 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca" pod "81e63562-ae59-4132-81d2-4389507b540e" (UID: "81e63562-ae59-4132-81d2-4389507b540e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:30.805241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.805213 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca" pod "81e63562-ae59-4132-81d2-4389507b540e" (UID: "81e63562-ae59-4132-81d2-4389507b540e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:03:30.805502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.805471 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw" pod "81e63562-ae59-4132-81d2-4389507b540e" (UID: "81e63562-ae59-4132-81d2-4389507b540e"). InnerVolumeSpecName "kube-api-access-22kxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:30.903798 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.903756 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e63562-ae59-4132-81d2-4389507b540e-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:30.903798 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.903801 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e63562-ae59-4132-81d2-4389507b540e-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:30.904228 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:30.903817 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22kxw\" (UniqueName: \"kubernetes.io/projected/81e63562-ae59-4132-81d2-4389507b540e-kube-api-access-22kxw\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:31.459536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:31.459485 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" event={"ID":"81e63562-ae59-4132-81d2-4389507b540e","Type":"ContainerDied","Data":"07eaec2b90581c12dab27f90a80abd9c616f71092c815b0274609f49c1be606e"} Oct 31 19:03:31.459759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:31.459593 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07eaec2b90581c12dab27f90a80abd9c616f71092c815b0274609f49c1be606e" Oct 31 19:03:31.459759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:31.459669 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs" Oct 31 19:03:33.784754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.784721 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7"] Oct 31 19:03:33.785226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.785194 5688 cpu_manager.go:414] "containerMap: removing container" podUID="81e63562-ae59-4132-81d2-4389507b540e" containerName="revoke-keys" Oct 31 19:03:33.785226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.785210 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e63562-ae59-4132-81d2-4389507b540e" containerName="revoke-keys" Oct 31 19:03:33.785330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.785320 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e63562-ae59-4132-81d2-4389507b540e" containerName="revoke-keys" Oct 31 19:03:33.788976 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.788948 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.839645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.839619 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.839803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.839655 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.839803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.839677 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntcp\" (UniqueName: \"kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.849048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.849008 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7"] Oct 31 19:03:33.890844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.890812 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:03:33.891052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.890823 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:03:33.940795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.940760 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.941002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.940802 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.941002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.940820 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntcp\" (UniqueName: \"kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.941477 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.941449 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.943565 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.943508 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:33.997397 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:33.997371 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntcp\" (UniqueName: \"kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp\") pod \"maas-api-revoke-sub-6971358f-mq2k7\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:34.102234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:34.102140 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:34.262336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:34.262302 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg"] Oct 31 19:03:34.262556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:34.262529 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" containerID="cri-o://67bd70806d3d356848fa48c66fd9e2a56497d9fe3b4a27562e83a26eff0a594f" gracePeriod=30 Oct 31 19:03:34.269232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:34.269205 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d"] Oct 31 19:03:34.269421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:34.269400 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" containerID="cri-o://e42e17b4e94048608a778fafdb77b6aace7a1fa8d11e1152430c77ce78d44eab" gracePeriod=30 Oct 31 19:03:34.539578 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:34.539552 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7"] Oct 31 19:03:34.542151 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:34.542123 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8799857_4376_4dc8_b27b_1abf614320fa.slice/crio-de17fc6c0c1c2ae74fd7e7bfe9449b9ffc3e3cee9842db90cce94102767d6c9d WatchSource:0}: Error finding container de17fc6c0c1c2ae74fd7e7bfe9449b9ffc3e3cee9842db90cce94102767d6c9d: Status 404 returned error can't find the container with id de17fc6c0c1c2ae74fd7e7bfe9449b9ffc3e3cee9842db90cce94102767d6c9d Oct 31 19:03:35.490870 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:35.490829 5688 generic.go:356] "Generic (PLEG): container finished" podID="f8799857-4376-4dc8-b27b-1abf614320fa" containerID="cccf319a521cb31cfde357f5b1adcdee7a48162b5aa19c624a6ca108ac1766cb" exitCode=0 Oct 31 19:03:35.491270 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:35.490921 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" event={"ID":"f8799857-4376-4dc8-b27b-1abf614320fa","Type":"ContainerDied","Data":"cccf319a521cb31cfde357f5b1adcdee7a48162b5aa19c624a6ca108ac1766cb"} Oct 31 19:03:35.491270 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:35.490993 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" event={"ID":"f8799857-4376-4dc8-b27b-1abf614320fa","Type":"ContainerStarted","Data":"de17fc6c0c1c2ae74fd7e7bfe9449b9ffc3e3cee9842db90cce94102767d6c9d"} Oct 31 19:03:36.694267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.694241 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:36.771560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.771467 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp\" (UniqueName: \"kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp\") pod \"f8799857-4376-4dc8-b27b-1abf614320fa\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " Oct 31 19:03:36.771560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.771543 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca\") pod \"f8799857-4376-4dc8-b27b-1abf614320fa\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " Oct 31 19:03:36.771792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.771597 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca\") pod \"f8799857-4376-4dc8-b27b-1abf614320fa\" (UID: \"f8799857-4376-4dc8-b27b-1abf614320fa\") " Oct 31 19:03:36.772048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.772022 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca" pod "f8799857-4376-4dc8-b27b-1abf614320fa" (UID: "f8799857-4376-4dc8-b27b-1abf614320fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:36.772168 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.772138 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:36.773866 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.773836 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca" pod "f8799857-4376-4dc8-b27b-1abf614320fa" (UID: "f8799857-4376-4dc8-b27b-1abf614320fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:03:36.774024 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.774005 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp" pod "f8799857-4376-4dc8-b27b-1abf614320fa" (UID: "f8799857-4376-4dc8-b27b-1abf614320fa"). InnerVolumeSpecName "kube-api-access-5ntcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:36.873660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.873624 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8799857-4376-4dc8-b27b-1abf614320fa-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:36.873660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:36.873654 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntcp\" (UniqueName: \"kubernetes.io/projected/f8799857-4376-4dc8-b27b-1abf614320fa-kube-api-access-5ntcp\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:37.504285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:37.504238 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" event={"ID":"f8799857-4376-4dc8-b27b-1abf614320fa","Type":"ContainerDied","Data":"de17fc6c0c1c2ae74fd7e7bfe9449b9ffc3e3cee9842db90cce94102767d6c9d"} Oct 31 19:03:37.504488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:37.504320 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de17fc6c0c1c2ae74fd7e7bfe9449b9ffc3e3cee9842db90cce94102767d6c9d" Oct 31 19:03:37.504488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:37.504258 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7" Oct 31 19:03:39.517167 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.517124 5688 generic.go:356] "Generic (PLEG): container finished" podID="ce634413-5f51-4efb-9370-f9812f2473ba" containerID="67bd70806d3d356848fa48c66fd9e2a56497d9fe3b4a27562e83a26eff0a594f" exitCode=0 Oct 31 19:03:39.517513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.517215 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" event={"ID":"ce634413-5f51-4efb-9370-f9812f2473ba","Type":"ContainerDied","Data":"67bd70806d3d356848fa48c66fd9e2a56497d9fe3b4a27562e83a26eff0a594f"} Oct 31 19:03:39.517513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.517289 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" event={"ID":"ce634413-5f51-4efb-9370-f9812f2473ba","Type":"ContainerDied","Data":"2f0d3173e8e80db8ff5404c181471a282ab21381c44fa30e3f37825696525416"} Oct 31 19:03:39.517513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.517306 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f0d3173e8e80db8ff5404c181471a282ab21381c44fa30e3f37825696525416" Oct 31 19:03:39.519066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.519045 5688 generic.go:356] "Generic (PLEG): container finished" podID="0b137635-a697-45c9-8196-cd213f057873" containerID="e42e17b4e94048608a778fafdb77b6aace7a1fa8d11e1152430c77ce78d44eab" exitCode=0 Oct 31 19:03:39.519169 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.519081 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" event={"ID":"0b137635-a697-45c9-8196-cd213f057873","Type":"ContainerDied","Data":"e42e17b4e94048608a778fafdb77b6aace7a1fa8d11e1152430c77ce78d44eab"} Oct 31 19:03:39.529729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.529678 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:03:39.554028 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.554008 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:03:39.605337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605311 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605351 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.605472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605369 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605423 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605455 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605483 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605582 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.605687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605616 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.605687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605635 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605664 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605695 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605734 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605751 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605766 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605796 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605831 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605856 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.605948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605868 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.606411 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.605987 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.606411 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.606081 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:39.606592 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.606550 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.606592 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.606601 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.606833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.606637 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.606833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.606673 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy\") pod \"0b137635-a697-45c9-8196-cd213f057873\" (UID: \"0b137635-a697-45c9-8196-cd213f057873\") " Oct 31 19:03:39.606833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.606725 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs\") pod \"ce634413-5f51-4efb-9370-f9812f2473ba\" (UID: \"ce634413-5f51-4efb-9370-f9812f2473ba\") " Oct 31 19:03:39.607252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607230 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce634413-5f51-4efb-9370-f9812f2473ba-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607255 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607269 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607281 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607298 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607312 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607326 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.607811 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.607791 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.608106 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.608087 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:39.608311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.608291 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "kube-api-access-qjbtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:39.608393 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.608327 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.608393 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.608337 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "kube-api-access-d8p6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:39.608916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.608576 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.608916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.608664 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:39.609251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.609231 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:39.609251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.609238 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo" pod "ce634413-5f51-4efb-9370-f9812f2473ba" (UID: "ce634413-5f51-4efb-9370-f9812f2473ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:03:39.609844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.609818 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:03:39.610687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.610669 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy" pod "0b137635-a697-45c9-8196-cd213f057873" (UID: "0b137635-a697-45c9-8196-cd213f057873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:03:39.708118 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708081 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b137635-a697-45c9-8196-cd213f057873-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708118 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708121 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708136 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708157 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708171 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce634413-5f51-4efb-9370-f9812f2473ba-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708201 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708214 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b137635-a697-45c9-8196-cd213f057873-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708227 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b137635-a697-45c9-8196-cd213f057873-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708240 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce634413-5f51-4efb-9370-f9812f2473ba-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708254 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8p6x\" (UniqueName: \"kubernetes.io/projected/ce634413-5f51-4efb-9370-f9812f2473ba-kube-api-access-d8p6x\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:39.708331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:39.708270 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjbtq\" (UniqueName: \"kubernetes.io/projected/0b137635-a697-45c9-8196-cd213f057873-kube-api-access-qjbtq\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:40.525342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.525300 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" Oct 31 19:03:40.525342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.525332 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d" event={"ID":"0b137635-a697-45c9-8196-cd213f057873","Type":"ContainerDied","Data":"dc7efa1e350363dcf20ea8876990b6a5ac12fbdd96cb931397f362a4ba231db9"} Oct 31 19:03:40.525841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.525390 5688 scope.go:122] "RemoveContainer" containerID="e42e17b4e94048608a778fafdb77b6aace7a1fa8d11e1152430c77ce78d44eab" Oct 31 19:03:40.525841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.525321 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg" Oct 31 19:03:40.687880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.687847 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d"] Oct 31 19:03:40.745206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.738909 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-h7s9d"] Oct 31 19:03:40.798553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.798470 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg"] Oct 31 19:03:40.803039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:40.803011 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-q2dbg"] Oct 31 19:03:41.566401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:41.566370 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b137635-a697-45c9-8196-cd213f057873" path="/var/lib/kubelet/pods/0b137635-a697-45c9-8196-cd213f057873/volumes" Oct 31 19:03:41.566840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:41.566825 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" path="/var/lib/kubelet/pods/ce634413-5f51-4efb-9370-f9812f2473ba/volumes" Oct 31 19:03:43.143046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143011 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9"] Oct 31 19:03:43.143532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143434 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" Oct 31 19:03:43.143532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143446 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" Oct 31 19:03:43.143532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143465 5688 cpu_manager.go:414] "containerMap: removing container" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" Oct 31 19:03:43.143532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143470 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" Oct 31 19:03:43.143532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143492 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f8799857-4376-4dc8-b27b-1abf614320fa" containerName="revoke-keys" Oct 31 19:03:43.143532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143497 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8799857-4376-4dc8-b27b-1abf614320fa" containerName="revoke-keys" Oct 31 19:03:43.143730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143562 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8799857-4376-4dc8-b27b-1abf614320fa" containerName="revoke-keys" Oct 31 19:03:43.143730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143569 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce634413-5f51-4efb-9370-f9812f2473ba" containerName="istio-proxy" Oct 31 19:03:43.143730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.143580 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b137635-a697-45c9-8196-cd213f057873" containerName="istio-proxy" Oct 31 19:03:43.148060 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.148037 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.204905 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.204870 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9"] Oct 31 19:03:43.246120 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.246089 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.246314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.246161 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.246362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.246312 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw72m\" (UniqueName: \"kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.250216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.250172 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:03:43.250345 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.250286 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:03:43.347279 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.347244 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw72m\" (UniqueName: \"kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.347468 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.347338 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.347468 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.347393 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.347970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.347951 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.349896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.349870 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.404026 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.403945 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw72m\" (UniqueName: \"kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m\") pod \"maas-api-revoke-sub-a5555b8a-klct9\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.461459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.461425 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:43.834417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:43.834389 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9"] Oct 31 19:03:43.836379 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:43.836338 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c8f1dd_2186_47b4_a074_5b18afcec560.slice/crio-65d3415913427969836d3b7a2f998fabd72aeec2111de49bf94804f5422a3f47 WatchSource:0}: Error finding container 65d3415913427969836d3b7a2f998fabd72aeec2111de49bf94804f5422a3f47: Status 404 returned error can't find the container with id 65d3415913427969836d3b7a2f998fabd72aeec2111de49bf94804f5422a3f47 Oct 31 19:03:44.549983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:44.549945 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" event={"ID":"83c8f1dd-2186-47b4-a074-5b18afcec560","Type":"ContainerStarted","Data":"4c2b251075874e3336963c34adeeddbfbd96d90cd9bdcadd7813010a13e1dc94"} Oct 31 19:03:44.550489 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:44.550012 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" event={"ID":"83c8f1dd-2186-47b4-a074-5b18afcec560","Type":"ContainerStarted","Data":"65d3415913427969836d3b7a2f998fabd72aeec2111de49bf94804f5422a3f47"} Oct 31 19:03:44.663648 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:44.663595 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" podStartSLOduration=1.663581889 podStartE2EDuration="1.663581889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:44.661527605 +0000 UTC m=+2545.263668748" watchObservedRunningTime="2026-10-31 19:03:44.663581889 +0000 UTC m=+2545.265723061" Oct 31 19:03:45.557242 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:45.557209 5688 generic.go:356] "Generic (PLEG): container finished" podID="83c8f1dd-2186-47b4-a074-5b18afcec560" containerID="4c2b251075874e3336963c34adeeddbfbd96d90cd9bdcadd7813010a13e1dc94" exitCode=0 Oct 31 19:03:45.557647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:45.557302 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" event={"ID":"83c8f1dd-2186-47b4-a074-5b18afcec560","Type":"ContainerDied","Data":"4c2b251075874e3336963c34adeeddbfbd96d90cd9bdcadd7813010a13e1dc94"} Oct 31 19:03:46.711933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.711906 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:46.784483 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.784451 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca\") pod \"83c8f1dd-2186-47b4-a074-5b18afcec560\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " Oct 31 19:03:46.784670 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.784490 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m\" (UniqueName: \"kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m\") pod \"83c8f1dd-2186-47b4-a074-5b18afcec560\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " Oct 31 19:03:46.784670 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.784531 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca\") pod \"83c8f1dd-2186-47b4-a074-5b18afcec560\" (UID: \"83c8f1dd-2186-47b4-a074-5b18afcec560\") " Oct 31 19:03:46.784912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.784892 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca" pod "83c8f1dd-2186-47b4-a074-5b18afcec560" (UID: "83c8f1dd-2186-47b4-a074-5b18afcec560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:46.784978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.784963 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:46.786926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.786895 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca" pod "83c8f1dd-2186-47b4-a074-5b18afcec560" (UID: "83c8f1dd-2186-47b4-a074-5b18afcec560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:03:46.787066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.786942 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m" pod "83c8f1dd-2186-47b4-a074-5b18afcec560" (UID: "83c8f1dd-2186-47b4-a074-5b18afcec560"). InnerVolumeSpecName "kube-api-access-pw72m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:46.885958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.885873 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw72m\" (UniqueName: \"kubernetes.io/projected/83c8f1dd-2186-47b4-a074-5b18afcec560-kube-api-access-pw72m\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:46.885958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:46.885908 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c8f1dd-2186-47b4-a074-5b18afcec560-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:47.569253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:47.569223 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" Oct 31 19:03:47.569253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:47.569237 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9" event={"ID":"83c8f1dd-2186-47b4-a074-5b18afcec560","Type":"ContainerDied","Data":"65d3415913427969836d3b7a2f998fabd72aeec2111de49bf94804f5422a3f47"} Oct 31 19:03:47.569464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:47.569293 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d3415913427969836d3b7a2f998fabd72aeec2111de49bf94804f5422a3f47" Oct 31 19:03:49.275108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.275069 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw"] Oct 31 19:03:49.275778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.275756 5688 cpu_manager.go:414] "containerMap: removing container" podUID="83c8f1dd-2186-47b4-a074-5b18afcec560" containerName="revoke-keys" Oct 31 19:03:49.275825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.275782 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c8f1dd-2186-47b4-a074-5b18afcec560" containerName="revoke-keys" Oct 31 19:03:49.275903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.275890 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c8f1dd-2186-47b4-a074-5b18afcec560" containerName="revoke-keys" Oct 31 19:03:49.279686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.279670 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.335573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.335538 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw"] Oct 31 19:03:49.381745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.381715 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:03:49.381939 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.381914 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:03:49.425317 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.425285 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw448\" (UniqueName: \"kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.425518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.425325 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.425518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.425370 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.526745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.526656 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw448\" (UniqueName: \"kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.526745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.526699 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.526745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.526734 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.527442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.527415 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.529239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.529219 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.583599 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.583567 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw448\" (UniqueName: \"kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448\") pod \"maas-api-revoke-sub-10ef2e4a-7fllw\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.593282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.593253 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:49.985671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:49.985635 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw"] Oct 31 19:03:49.987917 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:49.987891 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6818c607_4187_4e15_a779_53bd782cdef8.slice/crio-6d5064e8558016fb67f3f930a9de2cdbe1eab9d81719fa6d5684a6cab3e0d5fd WatchSource:0}: Error finding container 6d5064e8558016fb67f3f930a9de2cdbe1eab9d81719fa6d5684a6cab3e0d5fd: Status 404 returned error can't find the container with id 6d5064e8558016fb67f3f930a9de2cdbe1eab9d81719fa6d5684a6cab3e0d5fd Oct 31 19:03:50.586915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:50.586834 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" event={"ID":"6818c607-4187-4e15-a779-53bd782cdef8","Type":"ContainerStarted","Data":"c08371964ec339369c42a1077188a382d818be7e1fe625bd3410078bbbbfc137"} Oct 31 19:03:50.586915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:50.586886 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" event={"ID":"6818c607-4187-4e15-a779-53bd782cdef8","Type":"ContainerStarted","Data":"6d5064e8558016fb67f3f930a9de2cdbe1eab9d81719fa6d5684a6cab3e0d5fd"} Oct 31 19:03:50.703653 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:50.703604 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" podStartSLOduration=1.703591332 podStartE2EDuration="1.703591332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:50.701841757 +0000 UTC m=+2551.303982901" watchObservedRunningTime="2026-10-31 19:03:50.703591332 +0000 UTC m=+2551.305732476" Oct 31 19:03:51.593828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:51.593791 5688 generic.go:356] "Generic (PLEG): container finished" podID="6818c607-4187-4e15-a779-53bd782cdef8" containerID="c08371964ec339369c42a1077188a382d818be7e1fe625bd3410078bbbbfc137" exitCode=0 Oct 31 19:03:51.594245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:51.593873 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" event={"ID":"6818c607-4187-4e15-a779-53bd782cdef8","Type":"ContainerDied","Data":"c08371964ec339369c42a1077188a382d818be7e1fe625bd3410078bbbbfc137"} Oct 31 19:03:52.751327 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.751301 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:52.861995 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.861903 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca\") pod \"6818c607-4187-4e15-a779-53bd782cdef8\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " Oct 31 19:03:52.862212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.862034 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448\" (UniqueName: \"kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448\") pod \"6818c607-4187-4e15-a779-53bd782cdef8\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " Oct 31 19:03:52.862212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.862139 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca\") pod \"6818c607-4187-4e15-a779-53bd782cdef8\" (UID: \"6818c607-4187-4e15-a779-53bd782cdef8\") " Oct 31 19:03:52.862556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.862526 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca" pod "6818c607-4187-4e15-a779-53bd782cdef8" (UID: "6818c607-4187-4e15-a779-53bd782cdef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:03:52.864861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.864831 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca" pod "6818c607-4187-4e15-a779-53bd782cdef8" (UID: "6818c607-4187-4e15-a779-53bd782cdef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:03:52.865006 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.864949 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448" pod "6818c607-4187-4e15-a779-53bd782cdef8" (UID: "6818c607-4187-4e15-a779-53bd782cdef8"). InnerVolumeSpecName "kube-api-access-hw448". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:03:52.963562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.963518 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:52.963562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.963556 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6818c607-4187-4e15-a779-53bd782cdef8-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:52.963562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:52.963569 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw448\" (UniqueName: \"kubernetes.io/projected/6818c607-4187-4e15-a779-53bd782cdef8-kube-api-access-hw448\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:03:53.606508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:53.606478 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" Oct 31 19:03:53.606676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:53.606519 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw" event={"ID":"6818c607-4187-4e15-a779-53bd782cdef8","Type":"ContainerDied","Data":"6d5064e8558016fb67f3f930a9de2cdbe1eab9d81719fa6d5684a6cab3e0d5fd"} Oct 31 19:03:53.606676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:53.606563 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d5064e8558016fb67f3f930a9de2cdbe1eab9d81719fa6d5684a6cab3e0d5fd" Oct 31 19:03:54.904828 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:03:54.904790 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:03:57.825424 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.825389 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq"] Oct 31 19:03:57.825863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.825807 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6818c607-4187-4e15-a779-53bd782cdef8" containerName="revoke-keys" Oct 31 19:03:57.825863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.825817 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6818c607-4187-4e15-a779-53bd782cdef8" containerName="revoke-keys" Oct 31 19:03:57.825945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.825914 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6818c607-4187-4e15-a779-53bd782cdef8" containerName="revoke-keys" Oct 31 19:03:57.829270 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.829245 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918223 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzcv\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918268 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918295 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918429 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918550 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918508 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918585 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918550 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918597 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918714 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918699 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:57.918844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:57.918820 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020073 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020035 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzcv\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020073 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020079 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020095 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020117 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020143 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020213 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020257 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020293 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020347 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020670 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020643 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020684 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020756 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020867 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020845 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.020939 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.020924 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.022713 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.022691 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.022909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.022892 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.035156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.035130 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq"] Oct 31 19:03:58.076889 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.076814 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.077021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.076935 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzcv\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.143355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.143320 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:03:58.576949 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.576920 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq"] Oct 31 19:03:58.578770 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:03:58.578741 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4304df_7825_46d7_81f0_83d1f789061f.slice/crio-03b88f0af96e9dccb309cf9b7acd05f39c7952828f744540370bce954b85dc66 WatchSource:0}: Error finding container 03b88f0af96e9dccb309cf9b7acd05f39c7952828f744540370bce954b85dc66: Status 404 returned error can't find the container with id 03b88f0af96e9dccb309cf9b7acd05f39c7952828f744540370bce954b85dc66 Oct 31 19:03:58.581071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.581037 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:03:58.581132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.581105 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:03:58.581168 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.581135 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:03:58.634596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:58.634562 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" event={"ID":"1f4304df-7825-46d7-81f0-83d1f789061f","Type":"ContainerStarted","Data":"03b88f0af96e9dccb309cf9b7acd05f39c7952828f744540370bce954b85dc66"} Oct 31 19:03:59.664440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:59.664393 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" event={"ID":"1f4304df-7825-46d7-81f0-83d1f789061f","Type":"ContainerStarted","Data":"cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a"} Oct 31 19:03:59.884407 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:03:59.884358 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podStartSLOduration=2.884342354 podStartE2EDuration="2.884342354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:59.837501656 +0000 UTC m=+2560.439642829" watchObservedRunningTime="2026-10-31 19:03:59.884342354 +0000 UTC m=+2560.486483572" Oct 31 19:04:00.153893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:00.153858 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:04:00.154255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:00.154233 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 19:04:00.154478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:00.154454 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 19:04:01.146580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:01.146543 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 19:04:01.147215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:01.147156 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 19:04:02.147606 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:02.147566 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 19:04:02.148243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:02.148166 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 19:04:03.151210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:03.147500 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 19:04:03.151210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:03.147627 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 19:04:04.148618 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.148577 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 19:04:04.148904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.148873 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 19:04:04.353587 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.353542 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c"] Oct 31 19:04:04.358779 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.358751 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.410649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.410569 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c"] Oct 31 19:04:04.413492 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.413460 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrht\" (UniqueName: \"kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.413642 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.413594 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.413853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.413826 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.416052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.416028 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:04:04.456871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.456840 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:04:04.515162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.515129 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.515393 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.515171 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrht\" (UniqueName: \"kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.515393 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.515239 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.515852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.515831 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.517678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.517657 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.571871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.571833 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrht\" (UniqueName: \"kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht\") pod \"maas-api-revoke-sub-25845886-n797c\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:04.674937 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:04.674846 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:05.057210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:05.056856 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c"] Oct 31 19:04:05.059527 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:05.059493 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1408903_0386_47d7_b9c6_ac0435eea1b9.slice/crio-0bd42c0916991e370503a18b8f4885d144624cb9563d4bed3f8864823f4c1477 WatchSource:0}: Error finding container 0bd42c0916991e370503a18b8f4885d144624cb9563d4bed3f8864823f4c1477: Status 404 returned error can't find the container with id 0bd42c0916991e370503a18b8f4885d144624cb9563d4bed3f8864823f4c1477 Oct 31 19:04:05.144528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:05.144492 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 19:04:05.144725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:05.144554 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 19:04:05.706628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:05.706538 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" event={"ID":"f1408903-0386-47d7-b9c6-ac0435eea1b9","Type":"ContainerStarted","Data":"ab6cd37b63c04881c8c60fd313b9ae2ac65584102eea8dd29e0fad90e61eae12"} Oct 31 19:04:05.706628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:05.706602 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" event={"ID":"f1408903-0386-47d7-b9c6-ac0435eea1b9","Type":"ContainerStarted","Data":"0bd42c0916991e370503a18b8f4885d144624cb9563d4bed3f8864823f4c1477"} Oct 31 19:04:05.821125 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:05.821069 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" podStartSLOduration=1.8210544560000002 podStartE2EDuration="1.821054456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:05.818659503 +0000 UTC m=+2566.420800645" watchObservedRunningTime="2026-10-31 19:04:05.821054456 +0000 UTC m=+2566.423195600" Oct 31 19:04:06.148654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:06.148622 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:04:06.149043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:06.149024 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:04:06.149800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:06.149779 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:04:06.713312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:06.713275 5688 generic.go:356] "Generic (PLEG): container finished" podID="f1408903-0386-47d7-b9c6-ac0435eea1b9" containerID="ab6cd37b63c04881c8c60fd313b9ae2ac65584102eea8dd29e0fad90e61eae12" exitCode=0 Oct 31 19:04:06.713695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:06.713361 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" event={"ID":"f1408903-0386-47d7-b9c6-ac0435eea1b9","Type":"ContainerDied","Data":"ab6cd37b63c04881c8c60fd313b9ae2ac65584102eea8dd29e0fad90e61eae12"} Oct 31 19:04:07.869010 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.868984 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:07.959260 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.959227 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca\") pod \"f1408903-0386-47d7-b9c6-ac0435eea1b9\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " Oct 31 19:04:07.959455 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.959270 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht\" (UniqueName: \"kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht\") pod \"f1408903-0386-47d7-b9c6-ac0435eea1b9\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " Oct 31 19:04:07.959455 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.959343 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca\") pod \"f1408903-0386-47d7-b9c6-ac0435eea1b9\" (UID: \"f1408903-0386-47d7-b9c6-ac0435eea1b9\") " Oct 31 19:04:07.959632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.959604 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca" pod "f1408903-0386-47d7-b9c6-ac0435eea1b9" (UID: "f1408903-0386-47d7-b9c6-ac0435eea1b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:04:07.961617 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.961589 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca" pod "f1408903-0386-47d7-b9c6-ac0435eea1b9" (UID: "f1408903-0386-47d7-b9c6-ac0435eea1b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:04:07.961617 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:07.961595 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht" pod "f1408903-0386-47d7-b9c6-ac0435eea1b9" (UID: "f1408903-0386-47d7-b9c6-ac0435eea1b9"). InnerVolumeSpecName "kube-api-access-zcrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:04:08.060963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:08.060927 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcrht\" (UniqueName: \"kubernetes.io/projected/f1408903-0386-47d7-b9c6-ac0435eea1b9-kube-api-access-zcrht\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:08.060963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:08.060959 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:08.060963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:08.060969 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1408903-0386-47d7-b9c6-ac0435eea1b9-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:08.726757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:08.726725 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" event={"ID":"f1408903-0386-47d7-b9c6-ac0435eea1b9","Type":"ContainerDied","Data":"0bd42c0916991e370503a18b8f4885d144624cb9563d4bed3f8864823f4c1477"} Oct 31 19:04:08.726924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:08.726767 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c" Oct 31 19:04:08.726924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:08.726772 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd42c0916991e370503a18b8f4885d144624cb9563d4bed3f8864823f4c1477" Oct 31 19:04:11.093173 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.093129 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv"] Oct 31 19:04:11.093826 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.093803 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f1408903-0386-47d7-b9c6-ac0435eea1b9" containerName="revoke-keys" Oct 31 19:04:11.093928 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.093827 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1408903-0386-47d7-b9c6-ac0435eea1b9" containerName="revoke-keys" Oct 31 19:04:11.094009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.093996 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1408903-0386-47d7-b9c6-ac0435eea1b9" containerName="revoke-keys" Oct 31 19:04:11.097473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.097449 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.151145 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.151113 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:04:11.191375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.191341 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4wd\" (UniqueName: \"kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.191533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.191507 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.191595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.191575 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.200661 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.200639 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:04:11.203196 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.203157 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv"] Oct 31 19:04:11.293015 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.292981 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4wd\" (UniqueName: \"kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.293173 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.293093 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.293173 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.293121 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.293748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.293724 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.295523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.295503 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.349673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.349605 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4wd\" (UniqueName: \"kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd\") pod \"maas-api-revoke-sub-cf471b15-z5xxv\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.410954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.410923 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:11.787301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:11.787268 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv"] Oct 31 19:04:11.788925 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:11.788896 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726b2d22_a390_4532_b372_3f3324e71112.slice/crio-d9b1f8cd737d67d4bb8bbd868c3202e8a2bbf46890214a9d822824ac303d0ef5 WatchSource:0}: Error finding container d9b1f8cd737d67d4bb8bbd868c3202e8a2bbf46890214a9d822824ac303d0ef5: Status 404 returned error can't find the container with id d9b1f8cd737d67d4bb8bbd868c3202e8a2bbf46890214a9d822824ac303d0ef5 Oct 31 19:04:12.750612 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:12.750575 5688 generic.go:356] "Generic (PLEG): container finished" podID="726b2d22-a390-4532-b372-3f3324e71112" containerID="88789acfa2b8f5ee4501f1583ef80b63d83a9d21575ae5cf1e6cd0fbc3e90618" exitCode=0 Oct 31 19:04:12.751027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:12.750658 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" event={"ID":"726b2d22-a390-4532-b372-3f3324e71112","Type":"ContainerDied","Data":"88789acfa2b8f5ee4501f1583ef80b63d83a9d21575ae5cf1e6cd0fbc3e90618"} Oct 31 19:04:12.751027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:12.750713 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" event={"ID":"726b2d22-a390-4532-b372-3f3324e71112","Type":"ContainerStarted","Data":"d9b1f8cd737d67d4bb8bbd868c3202e8a2bbf46890214a9d822824ac303d0ef5"} Oct 31 19:04:13.902426 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:13.902400 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:14.023052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.022959 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca\") pod \"726b2d22-a390-4532-b372-3f3324e71112\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " Oct 31 19:04:14.023234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.023050 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca\") pod \"726b2d22-a390-4532-b372-3f3324e71112\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " Oct 31 19:04:14.023234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.023143 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd\" (UniqueName: \"kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd\") pod \"726b2d22-a390-4532-b372-3f3324e71112\" (UID: \"726b2d22-a390-4532-b372-3f3324e71112\") " Oct 31 19:04:14.023767 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.023737 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca" pod "726b2d22-a390-4532-b372-3f3324e71112" (UID: "726b2d22-a390-4532-b372-3f3324e71112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:04:14.025683 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.025660 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca" pod "726b2d22-a390-4532-b372-3f3324e71112" (UID: "726b2d22-a390-4532-b372-3f3324e71112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:04:14.025954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.025927 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd" pod "726b2d22-a390-4532-b372-3f3324e71112" (UID: "726b2d22-a390-4532-b372-3f3324e71112"). InnerVolumeSpecName "kube-api-access-fm4wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:04:14.125161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.125129 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b2d22-a390-4532-b372-3f3324e71112-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:14.125161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.125159 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b2d22-a390-4532-b372-3f3324e71112-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:14.125161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.125169 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm4wd\" (UniqueName: \"kubernetes.io/projected/726b2d22-a390-4532-b372-3f3324e71112-kube-api-access-fm4wd\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:14.773196 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.767971 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" Oct 31 19:04:14.773196 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.768081 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv" event={"ID":"726b2d22-a390-4532-b372-3f3324e71112","Type":"ContainerDied","Data":"d9b1f8cd737d67d4bb8bbd868c3202e8a2bbf46890214a9d822824ac303d0ef5"} Oct 31 19:04:14.773196 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:14.768127 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b1f8cd737d67d4bb8bbd868c3202e8a2bbf46890214a9d822824ac303d0ef5" Oct 31 19:04:17.292910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.292875 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9"] Oct 31 19:04:17.293539 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.293519 5688 cpu_manager.go:414] "containerMap: removing container" podUID="726b2d22-a390-4532-b372-3f3324e71112" containerName="revoke-keys" Oct 31 19:04:17.293735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.293541 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726b2d22-a390-4532-b372-3f3324e71112" containerName="revoke-keys" Oct 31 19:04:17.293735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.293657 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="726b2d22-a390-4532-b372-3f3324e71112" containerName="revoke-keys" Oct 31 19:04:17.297021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.296998 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.350096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.350049 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:04:17.353111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.353082 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9"] Oct 31 19:04:17.373653 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.373622 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.373825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.373678 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27m85\" (UniqueName: \"kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.373895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.373859 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.399366 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.399339 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:04:17.474455 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.474425 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.474599 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.474494 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.474599 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.474537 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27m85\" (UniqueName: \"kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.475085 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.475060 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.476887 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.476869 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.530440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.530408 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27m85\" (UniqueName: \"kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85\") pod \"maas-api-revoke-sub-b366e594-dctl9\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.610355 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.610281 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:17.984349 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:17.984319 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9"] Oct 31 19:04:17.986648 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:17.986614 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74abb60a_d939_4ff7_9abb_650becb0fbc0.slice/crio-3ad2110cc531d5cb5a59cc7c83e18daed91eb6bce2e369cefb22fb17a9562b56 WatchSource:0}: Error finding container 3ad2110cc531d5cb5a59cc7c83e18daed91eb6bce2e369cefb22fb17a9562b56: Status 404 returned error can't find the container with id 3ad2110cc531d5cb5a59cc7c83e18daed91eb6bce2e369cefb22fb17a9562b56 Oct 31 19:04:18.795079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:18.795043 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" event={"ID":"74abb60a-d939-4ff7-9abb-650becb0fbc0","Type":"ContainerStarted","Data":"fa0460575f6c1786dae7cbe6ec2c252c25280921ef2df5f6d21032f118650a31"} Oct 31 19:04:18.795551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:18.795110 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" event={"ID":"74abb60a-d939-4ff7-9abb-650becb0fbc0","Type":"ContainerStarted","Data":"3ad2110cc531d5cb5a59cc7c83e18daed91eb6bce2e369cefb22fb17a9562b56"} Oct 31 19:04:18.908868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:18.908818 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" podStartSLOduration=1.908805643 podStartE2EDuration="1.908805643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:18.908360309 +0000 UTC m=+2579.510501452" watchObservedRunningTime="2026-10-31 19:04:18.908805643 +0000 UTC m=+2579.510946787" Oct 31 19:04:19.802997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:19.802964 5688 generic.go:356] "Generic (PLEG): container finished" podID="74abb60a-d939-4ff7-9abb-650becb0fbc0" containerID="fa0460575f6c1786dae7cbe6ec2c252c25280921ef2df5f6d21032f118650a31" exitCode=0 Oct 31 19:04:19.803528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:19.803044 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" event={"ID":"74abb60a-d939-4ff7-9abb-650becb0fbc0","Type":"ContainerDied","Data":"fa0460575f6c1786dae7cbe6ec2c252c25280921ef2df5f6d21032f118650a31"} Oct 31 19:04:20.966002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:20.965977 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:21.122143 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.122054 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca\") pod \"74abb60a-d939-4ff7-9abb-650becb0fbc0\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " Oct 31 19:04:21.122444 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.122424 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca\") pod \"74abb60a-d939-4ff7-9abb-650becb0fbc0\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " Oct 31 19:04:21.122596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.122578 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85\" (UniqueName: \"kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85\") pod \"74abb60a-d939-4ff7-9abb-650becb0fbc0\" (UID: \"74abb60a-d939-4ff7-9abb-650becb0fbc0\") " Oct 31 19:04:21.123768 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.123724 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca" pod "74abb60a-d939-4ff7-9abb-650becb0fbc0" (UID: "74abb60a-d939-4ff7-9abb-650becb0fbc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:04:21.125577 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.125543 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85" pod "74abb60a-d939-4ff7-9abb-650becb0fbc0" (UID: "74abb60a-d939-4ff7-9abb-650becb0fbc0"). InnerVolumeSpecName "kube-api-access-27m85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:04:21.126333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.126300 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca" pod "74abb60a-d939-4ff7-9abb-650becb0fbc0" (UID: "74abb60a-d939-4ff7-9abb-650becb0fbc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:04:21.224358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.224326 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:21.224358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.224355 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74abb60a-d939-4ff7-9abb-650becb0fbc0-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:21.224553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.224370 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27m85\" (UniqueName: \"kubernetes.io/projected/74abb60a-d939-4ff7-9abb-650becb0fbc0-kube-api-access-27m85\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:21.815215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.815169 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" Oct 31 19:04:21.815215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.815199 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9" event={"ID":"74abb60a-d939-4ff7-9abb-650becb0fbc0","Type":"ContainerDied","Data":"3ad2110cc531d5cb5a59cc7c83e18daed91eb6bce2e369cefb22fb17a9562b56"} Oct 31 19:04:21.815431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:21.815257 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ad2110cc531d5cb5a59cc7c83e18daed91eb6bce2e369cefb22fb17a9562b56" Oct 31 19:04:23.637194 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.637162 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7"] Oct 31 19:04:23.637638 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.637622 5688 cpu_manager.go:414] "containerMap: removing container" podUID="74abb60a-d939-4ff7-9abb-650becb0fbc0" containerName="revoke-keys" Oct 31 19:04:23.637692 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.637641 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74abb60a-d939-4ff7-9abb-650becb0fbc0" containerName="revoke-keys" Oct 31 19:04:23.637759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.637749 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="74abb60a-d939-4ff7-9abb-650becb0fbc0" containerName="revoke-keys" Oct 31 19:04:23.641359 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.641342 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650165 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650137 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650178 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650267 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650365 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650425 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650444 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650506 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650590 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.650822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.650624 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksml\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.692957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.692931 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-88085e-openshift-default-dockercfg-96c8d\"" Oct 31 19:04:23.746256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.746215 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7"] Oct 31 19:04:23.751880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.751846 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.751901 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.751942 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.751997 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752039 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksml\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752112 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752161 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752204 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752240 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752352 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752617 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.752769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.752662 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.753046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.753028 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.753311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.753284 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.755296 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.755272 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.755704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.755665 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.809036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.809000 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.809243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.809070 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksml\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:23.954780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:23.954701 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:24.404743 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.404716 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7"] Oct 31 19:04:24.406715 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:24.406686 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce1ee45_6e69_40b9_bb09_5f02629bcd1d.slice/crio-9e50c5bbf63c6ae8566ad634866e9130948cd2d999c9001d9ed6962778e7b34c WatchSource:0}: Error finding container 9e50c5bbf63c6ae8566ad634866e9130948cd2d999c9001d9ed6962778e7b34c: Status 404 returned error can't find the container with id 9e50c5bbf63c6ae8566ad634866e9130948cd2d999c9001d9ed6962778e7b34c Oct 31 19:04:24.409341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.409308 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:04:24.409433 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.409375 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:04:24.409433 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.409404 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:04:24.848066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.848028 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" event={"ID":"dce1ee45-6e69-40b9-bb09-5f02629bcd1d","Type":"ContainerStarted","Data":"a706097d45e4e26d70cd8bdbf6184a97749fc139b113fb97709f6da853db2b75"} Oct 31 19:04:24.848601 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.848095 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" event={"ID":"dce1ee45-6e69-40b9-bb09-5f02629bcd1d","Type":"ContainerStarted","Data":"9e50c5bbf63c6ae8566ad634866e9130948cd2d999c9001d9ed6962778e7b34c"} Oct 31 19:04:24.943134 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:04:24.943089 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:04:24.956252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:24.955365 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:25.109263 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:25.098978 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podStartSLOduration=2.098954837 podStartE2EDuration="2.098954837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:25.017266567 +0000 UTC m=+2585.619407710" watchObservedRunningTime="2026-10-31 19:04:25.098954837 +0000 UTC m=+2585.701095981" Oct 31 19:04:25.958381 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:25.958323 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 19:04:25.958921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:25.958394 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 19:04:26.961247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:26.957577 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 19:04:26.961247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:26.957657 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 19:04:27.957208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:27.956621 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 19:04:27.957208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:27.956686 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 19:04:28.027852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.027819 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp"] Oct 31 19:04:28.031616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.031595 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.083529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.083498 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\"" Oct 31 19:04:28.126774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.126734 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.126933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.126883 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.126976 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.126967 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm4m\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.127010 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.126991 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.127041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.127010 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.127077 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.127036 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.127109 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.127088 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.127141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.127108 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.127255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.127225 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.169105 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.169076 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp"] Oct 31 19:04:28.228299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228211 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228293 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm4m\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228317 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228337 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228363 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228494 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228534 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228569 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228585 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228648 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228734 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228649 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.228998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.228848 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.229131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.229118 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.230754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.230731 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.231132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.231111 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.285001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.284964 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.285154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.285070 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm4m\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.371442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.371404 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:28.808336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.808306 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp"] Oct 31 19:04:28.810137 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:28.810112 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c031e57_604b_42b5_8896_7798e48f749c.slice/crio-4dd606056991ca4ccdb6d468850d6b26b1bfdd6199c045a8b700ebfb01e8f4a1 WatchSource:0}: Error finding container 4dd606056991ca4ccdb6d468850d6b26b1bfdd6199c045a8b700ebfb01e8f4a1: Status 404 returned error can't find the container with id 4dd606056991ca4ccdb6d468850d6b26b1bfdd6199c045a8b700ebfb01e8f4a1 Oct 31 19:04:28.812539 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.812508 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:04:28.812621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.812578 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:04:28.812621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.812607 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:04:28.883470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.883441 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" event={"ID":"8c031e57-604b-42b5-8896-7798e48f749c","Type":"ContainerStarted","Data":"4dd606056991ca4ccdb6d468850d6b26b1bfdd6199c045a8b700ebfb01e8f4a1"} Oct 31 19:04:28.971225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.965825 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 19:04:28.971225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:28.965879 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 19:04:29.895239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:29.895174 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" event={"ID":"8c031e57-604b-42b5-8896-7798e48f749c","Type":"ContainerStarted","Data":"6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c"} Oct 31 19:04:29.958735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:29.958661 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 19:04:29.959228 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:29.958819 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 19:04:30.371916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:30.371865 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:30.373646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:30.373622 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:04:30.373774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:30.373666 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:04:30.955209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:30.955022 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 19:04:30.955209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:30.955081 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 19:04:31.371769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:31.371735 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:04:31.372014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:31.371983 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:04:31.962297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:31.962259 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:31.962845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:31.962820 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:31.963482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:31.963462 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:04:32.130407 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:32.130348 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podStartSLOduration=5.130329547 podStartE2EDuration="5.130329547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:30.140611488 +0000 UTC m=+2590.742752630" watchObservedRunningTime="2026-10-31 19:04:32.130329547 +0000 UTC m=+2592.732470692" Oct 31 19:04:32.372518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:32.372475 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:04:32.372730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:32.372544 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:04:33.372158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:33.372111 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:04:33.372574 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:33.372230 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:04:34.372116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.372074 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:04:34.372346 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.372139 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:04:34.811063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.811031 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9"] Oct 31 19:04:34.815302 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.815279 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.871400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.871361 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbq4q\" (UniqueName: \"kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.871697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.871677 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.871981 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.871957 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.872384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.872345 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9"] Oct 31 19:04:34.917521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.917474 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:04:34.917521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.917518 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:04:34.972796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.972758 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.972975 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.972809 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbq4q\" (UniqueName: \"kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.972975 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.972846 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.973447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.973426 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:34.975437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:34.975410 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:35.029634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.029600 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbq4q\" (UniqueName: \"kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q\") pod \"maas-api-revoke-sub-cae0ea62-75sl9\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:35.129136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.129032 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:35.372378 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.372346 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:04:35.372780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.372431 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:04:35.515162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.515135 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9"] Oct 31 19:04:35.517698 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:35.517669 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f3f080_2c7d_4d2f_9478_55e560f03542.slice/crio-9e4f86fb8f03aa819e0bdae6dcd6f4eca9e6f4c8b3e443430a766395f2e8f46e WatchSource:0}: Error finding container 9e4f86fb8f03aa819e0bdae6dcd6f4eca9e6f4c8b3e443430a766395f2e8f46e: Status 404 returned error can't find the container with id 9e4f86fb8f03aa819e0bdae6dcd6f4eca9e6f4c8b3e443430a766395f2e8f46e Oct 31 19:04:35.941285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.941238 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" event={"ID":"f6f3f080-2c7d-4d2f-9478-55e560f03542","Type":"ContainerStarted","Data":"1b4cee4a885537c7a23a1bcd8e414374080f943089329a55de50df5b214acbef"} Oct 31 19:04:35.941485 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:35.941302 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" event={"ID":"f6f3f080-2c7d-4d2f-9478-55e560f03542","Type":"ContainerStarted","Data":"9e4f86fb8f03aa819e0bdae6dcd6f4eca9e6f4c8b3e443430a766395f2e8f46e"} Oct 31 19:04:36.376472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:36.376445 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:36.376889 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:36.376869 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:36.377580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:36.377563 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:04:36.493816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:36.493754 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" podStartSLOduration=2.493738567 podStartE2EDuration="2.493738567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:36.054109851 +0000 UTC m=+2596.656250995" watchObservedRunningTime="2026-10-31 19:04:36.493738567 +0000 UTC m=+2597.095879753" Oct 31 19:04:36.948707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:36.948669 5688 generic.go:356] "Generic (PLEG): container finished" podID="f6f3f080-2c7d-4d2f-9478-55e560f03542" containerID="1b4cee4a885537c7a23a1bcd8e414374080f943089329a55de50df5b214acbef" exitCode=0 Oct 31 19:04:36.948970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:36.948754 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" event={"ID":"f6f3f080-2c7d-4d2f-9478-55e560f03542","Type":"ContainerDied","Data":"1b4cee4a885537c7a23a1bcd8e414374080f943089329a55de50df5b214acbef"} Oct 31 19:04:38.105091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.105061 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:38.119387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.119344 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q\" (UniqueName: \"kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q\") pod \"f6f3f080-2c7d-4d2f-9478-55e560f03542\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " Oct 31 19:04:38.120700 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.120676 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca\") pod \"f6f3f080-2c7d-4d2f-9478-55e560f03542\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " Oct 31 19:04:38.121956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.121936 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca\") pod \"f6f3f080-2c7d-4d2f-9478-55e560f03542\" (UID: \"f6f3f080-2c7d-4d2f-9478-55e560f03542\") " Oct 31 19:04:38.123301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.123173 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca" pod "f6f3f080-2c7d-4d2f-9478-55e560f03542" (UID: "f6f3f080-2c7d-4d2f-9478-55e560f03542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:04:38.124357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.124336 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:38.130746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.130482 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca" pod "f6f3f080-2c7d-4d2f-9478-55e560f03542" (UID: "f6f3f080-2c7d-4d2f-9478-55e560f03542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:04:38.131142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.131104 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q" pod "f6f3f080-2c7d-4d2f-9478-55e560f03542" (UID: "f6f3f080-2c7d-4d2f-9478-55e560f03542"). InnerVolumeSpecName "kube-api-access-hbq4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:04:38.227307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.227225 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f3f080-2c7d-4d2f-9478-55e560f03542-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:38.227527 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.227512 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbq4q\" (UniqueName: \"kubernetes.io/projected/f6f3f080-2c7d-4d2f-9478-55e560f03542-kube-api-access-hbq4q\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:38.961818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.961783 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" Oct 31 19:04:38.962003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.961812 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9" event={"ID":"f6f3f080-2c7d-4d2f-9478-55e560f03542","Type":"ContainerDied","Data":"9e4f86fb8f03aa819e0bdae6dcd6f4eca9e6f4c8b3e443430a766395f2e8f46e"} Oct 31 19:04:38.962003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:38.961883 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4f86fb8f03aa819e0bdae6dcd6f4eca9e6f4c8b3e443430a766395f2e8f46e" Oct 31 19:04:51.341444 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:51.341405 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq"] Oct 31 19:04:54.905422 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:04:54.905385 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:04:55.552678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.552638 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf"] Oct 31 19:04:55.553360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.553337 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f6f3f080-2c7d-4d2f-9478-55e560f03542" containerName="revoke-keys" Oct 31 19:04:55.553566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.553540 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f3f080-2c7d-4d2f-9478-55e560f03542" containerName="revoke-keys" Oct 31 19:04:55.553770 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.553757 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f3f080-2c7d-4d2f-9478-55e560f03542" containerName="revoke-keys" Oct 31 19:04:55.558066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.558046 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.602461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.602421 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf"] Oct 31 19:04:55.633879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.633839 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdcn\" (UniqueName: \"kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn\") pod \"payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.634126 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.633896 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.648538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.648516 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dbceb9\"" Oct 31 19:04:55.648774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.648755 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dbceb9-dockercfg-zqj24\"" Oct 31 19:04:55.734767 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.734730 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.734954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.734876 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdcn\" (UniqueName: \"kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn\") pod \"payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.735524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.735504 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.791428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.791391 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdcn\" (UniqueName: \"kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn\") pod \"payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:55.873630 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:55.873556 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:56.247373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.247347 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf"] Oct 31 19:04:56.249722 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:04:56.249692 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0ede4c_e957_4863_8939_0d7ef964c2f9.slice/crio-4da70cac734fdd265f414a4c162a634bb59799d38aedbf76273906f1a99217a3 WatchSource:0}: Error finding container 4da70cac734fdd265f414a4c162a634bb59799d38aedbf76273906f1a99217a3: Status 404 returned error can't find the container with id 4da70cac734fdd265f414a4c162a634bb59799d38aedbf76273906f1a99217a3 Oct 31 19:04:56.343161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.343101 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" podUID="1f8fb434-3b1c-4ab8-b600-3f42a319436b" containerName="payload-pre-processing" containerID="cri-o://dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9" gracePeriod=25 Oct 31 19:04:56.592374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.592346 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:04:56.643329 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.643300 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9\" (UniqueName: \"kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9\") pod \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " Oct 31 19:04:56.643484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.643352 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume\") pod \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\" (UID: \"1f8fb434-3b1c-4ab8-b600-3f42a319436b\") " Oct 31 19:04:56.643716 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.643695 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume" pod "1f8fb434-3b1c-4ab8-b600-3f42a319436b" (UID: "1f8fb434-3b1c-4ab8-b600-3f42a319436b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:04:56.645516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.645497 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9" pod "1f8fb434-3b1c-4ab8-b600-3f42a319436b" (UID: "1f8fb434-3b1c-4ab8-b600-3f42a319436b"). InnerVolumeSpecName "kube-api-access-blkz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:04:56.744491 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.744459 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkz9\" (UniqueName: \"kubernetes.io/projected/1f8fb434-3b1c-4ab8-b600-3f42a319436b-kube-api-access-blkz9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:56.744491 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:56.744486 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8fb434-3b1c-4ab8-b600-3f42a319436b-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:04:57.091236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.091117 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" event={"ID":"5d0ede4c-e957-4863-8939-0d7ef964c2f9","Type":"ContainerStarted","Data":"d3a13c83fc7d0c8b62f252f4241867730b1af71c642975114a32d75c96d37b33"} Oct 31 19:04:57.091236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.091177 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" event={"ID":"5d0ede4c-e957-4863-8939-0d7ef964c2f9","Type":"ContainerStarted","Data":"4da70cac734fdd265f414a4c162a634bb59799d38aedbf76273906f1a99217a3"} Oct 31 19:04:57.091465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.091257 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:04:57.092556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.092533 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f8fb434-3b1c-4ab8-b600-3f42a319436b" containerID="dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9" exitCode=0 Oct 31 19:04:57.092672 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.092568 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" Oct 31 19:04:57.092672 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.092565 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" event={"ID":"1f8fb434-3b1c-4ab8-b600-3f42a319436b","Type":"ContainerDied","Data":"dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9"} Oct 31 19:04:57.092759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.092683 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq" event={"ID":"1f8fb434-3b1c-4ab8-b600-3f42a319436b","Type":"ContainerDied","Data":"1c5632a345404b4a4ce676dd23afef27151f6826fe0089c7c98f3f368487719e"} Oct 31 19:04:57.092759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.092706 5688 scope.go:122] "RemoveContainer" containerID="dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9" Oct 31 19:04:57.102530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.102514 5688 scope.go:122] "RemoveContainer" containerID="dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9" Oct 31 19:04:57.102785 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:04:57.102763 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9\": container with ID starting with dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9 not found: ID does not exist" containerID="dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9" Oct 31 19:04:57.102853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.102792 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9"} err="failed to get container status \"dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9\": rpc error: code = NotFound desc = could not find container \"dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9\": container with ID starting with dcb1089aa36e3050aa646a001a816f48048690474155195c803d098f25ef53c9 not found: ID does not exist" Oct 31 19:04:57.205532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.205470 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" podStartSLOduration=2.205455503 podStartE2EDuration="2.205455503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:57.203601404 +0000 UTC m=+2617.805742547" watchObservedRunningTime="2026-10-31 19:04:57.205455503 +0000 UTC m=+2617.807596646" Oct 31 19:04:57.327158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.326988 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq"] Oct 31 19:04:57.327158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.327090 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-79e7b6-7d675fb6dc-jhzbq"] Oct 31 19:04:57.566332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:04:57.566303 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8fb434-3b1c-4ab8-b600-3f42a319436b" path="/var/lib/kubelet/pods/1f8fb434-3b1c-4ab8-b600-3f42a319436b/volumes" Oct 31 19:05:08.103394 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:08.103364 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:05:08.983287 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:08.983255 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m"] Oct 31 19:05:08.983723 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:08.983711 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f8fb434-3b1c-4ab8-b600-3f42a319436b" containerName="payload-pre-processing" Oct 31 19:05:08.983766 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:08.983725 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8fb434-3b1c-4ab8-b600-3f42a319436b" containerName="payload-pre-processing" Oct 31 19:05:08.983816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:08.983798 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8fb434-3b1c-4ab8-b600-3f42a319436b" containerName="payload-pre-processing" Oct 31 19:05:08.988800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:08.988772 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.090791 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.090760 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.090957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.090814 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.090957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.090846 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.090957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.090942 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.091148 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.090963 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.091280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.091155 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwbk\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.091280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.091224 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.091280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.091275 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.091443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.091317 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.093655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.093633 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m"] Oct 31 19:05:09.197984 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.197944 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwbk\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.199640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.199600 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.200280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.200258 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.200484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.200467 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.215528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.215500 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.216067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.216047 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.234381 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.234282 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.234677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.234659 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.234793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.234777 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.235046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.200633 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.235152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.215349 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.236150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.236116 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.236331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.200175 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.236423 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.215993 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.265384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.265230 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.269964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.269923 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.271643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.271614 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwbk\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.279388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.279358 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.302491 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.302460 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:09.731225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.731200 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m"] Oct 31 19:05:09.732749 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:05:09.732718 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7f3e8b_df19_4bbd_8e94_e47bb629298d.slice/crio-599f375f26a049a06acc512a995237ab80e7b2a1a3bb9b51f8fc7bba94cb2c68 WatchSource:0}: Error finding container 599f375f26a049a06acc512a995237ab80e7b2a1a3bb9b51f8fc7bba94cb2c68: Status 404 returned error can't find the container with id 599f375f26a049a06acc512a995237ab80e7b2a1a3bb9b51f8fc7bba94cb2c68 Oct 31 19:05:09.735244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.735174 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:05:09.735331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.735277 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:05:09.735331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:09.735304 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:05:10.221116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:10.221075 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" event={"ID":"1f7f3e8b-df19-4bbd-8e94-e47bb629298d","Type":"ContainerStarted","Data":"80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557"} Oct 31 19:05:10.221584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:10.221169 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" event={"ID":"1f7f3e8b-df19-4bbd-8e94-e47bb629298d","Type":"ContainerStarted","Data":"599f375f26a049a06acc512a995237ab80e7b2a1a3bb9b51f8fc7bba94cb2c68"} Oct 31 19:05:10.306616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:10.305340 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:10.317379 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:10.317331 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:10.317562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:10.317395 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:10.345208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:10.338521 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podStartSLOduration=2.33850407 podStartE2EDuration="2.33850407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:10.337878875 +0000 UTC m=+2630.940020019" watchObservedRunningTime="2026-10-31 19:05:10.33850407 +0000 UTC m=+2630.940645216" Oct 31 19:05:11.303085 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:11.303050 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:11.303478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:11.303109 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:12.302853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:12.302813 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:12.303048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:12.302871 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:12.966023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:12.965982 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7"] Oct 31 19:05:12.966600 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:12.966265 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" containerID="cri-o://53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159" gracePeriod=30 Oct 31 19:05:13.303442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:13.303399 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:13.303674 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:13.303463 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:14.303946 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:14.303571 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:14.303946 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:14.303621 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:15.302802 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:15.302763 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:15.302991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:15.302818 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:16.303380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:16.303346 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 19:05:16.303767 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:16.303404 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 19:05:17.307964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:17.307933 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:17.308442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:17.308424 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:17.309328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:17.309305 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:05:18.229661 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.229599 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:05:18.293448 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.293414 5688 generic.go:356] "Generic (PLEG): container finished" podID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerID="53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159" exitCode=0 Oct 31 19:05:18.293650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.293474 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" Oct 31 19:05:18.293650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.293495 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" event={"ID":"169b7f77-ea2e-4d64-925c-eb082102aa78","Type":"ContainerDied","Data":"53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159"} Oct 31 19:05:18.293650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.293546 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7" event={"ID":"169b7f77-ea2e-4d64-925c-eb082102aa78","Type":"ContainerDied","Data":"a33ac931f096eda4eeb0593fc9500b288cc6f0dc9d790c5197a9e2ad6fc02acf"} Oct 31 19:05:18.293650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.293574 5688 scope.go:122] "RemoveContainer" containerID="53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159" Oct 31 19:05:18.303153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.303135 5688 scope.go:122] "RemoveContainer" containerID="2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3" Oct 31 19:05:18.311850 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.311833 5688 scope.go:122] "RemoveContainer" containerID="53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159" Oct 31 19:05:18.312156 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:05:18.312075 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159\": container with ID starting with 53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159 not found: ID does not exist" containerID="53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159" Oct 31 19:05:18.312156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.312098 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159"} err="failed to get container status \"53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159\": rpc error: code = NotFound desc = could not find container \"53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159\": container with ID starting with 53bd61b63be2a1e7e39fd741ab705713b18f08ff9c5fbf73a312497db5df9159 not found: ID does not exist" Oct 31 19:05:18.312156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.312113 5688 scope.go:122] "RemoveContainer" containerID="2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3" Oct 31 19:05:18.312385 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:05:18.312368 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3\": container with ID starting with 2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3 not found: ID does not exist" containerID="2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3" Oct 31 19:05:18.312424 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.312390 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3"} err="failed to get container status \"2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3\": rpc error: code = NotFound desc = could not find container \"2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3\": container with ID starting with 2d4871da47eddc66329e8bdbcae8ea0f0dab8a976c754c1157dc935dde63b3c3 not found: ID does not exist" Oct 31 19:05:18.373363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373331 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373427 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373455 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373503 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373541 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373612 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373688 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373726 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373758 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:05:18.373989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373816 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo\") pod \"169b7f77-ea2e-4d64-925c-eb082102aa78\" (UID: \"169b7f77-ea2e-4d64-925c-eb082102aa78\") " Oct 31 19:05:18.373989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373910 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:05:18.373989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373981 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.373933 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.374242 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.374390 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.374413 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.374429 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.374443 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169b7f77-ea2e-4d64-925c-eb082102aa78-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.374457 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.376530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.376504 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:05:18.376676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.376649 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "kube-api-access-5mhtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:05:18.377583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.377555 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:05:18.377658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.377559 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token" pod "169b7f77-ea2e-4d64-925c-eb082102aa78" (UID: "169b7f77-ea2e-4d64-925c-eb082102aa78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:05:18.476092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.476058 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.476092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.476100 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.476333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.476116 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169b7f77-ea2e-4d64-925c-eb082102aa78-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.476333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.476129 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mhtk\" (UniqueName: \"kubernetes.io/projected/169b7f77-ea2e-4d64-925c-eb082102aa78-kube-api-access-5mhtk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:05:18.714533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.714496 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7"] Oct 31 19:05:18.720236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:18.720211 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-79e7b6-openshift-default-794798786f-hh4w7"] Oct 31 19:05:19.567741 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:19.567708 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" path="/var/lib/kubelet/pods/169b7f77-ea2e-4d64-925c-eb082102aa78/volumes" Oct 31 19:05:21.591071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.591035 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7"] Oct 31 19:05:21.591517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.591491 5688 cpu_manager.go:414] "containerMap: removing container" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" Oct 31 19:05:21.591517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.591503 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" Oct 31 19:05:21.591584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.591534 5688 cpu_manager.go:414] "containerMap: removing container" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" Oct 31 19:05:21.591584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.591540 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" Oct 31 19:05:21.591647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.591626 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" Oct 31 19:05:21.596572 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.596549 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:21.651317 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.651283 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7"] Oct 31 19:05:21.697551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.697524 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7006d3\"" Oct 31 19:05:21.789900 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.789863 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn"] Oct 31 19:05:21.791239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.791218 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="169b7f77-ea2e-4d64-925c-eb082102aa78" containerName="istio-proxy" Oct 31 19:05:21.800819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.800784 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:21.802234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.802172 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:21.802377 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.802292 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hlm\" (UniqueName: \"kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm\") pod \"payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:21.803751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.803724 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn"] Oct 31 19:05:21.853501 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.853419 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7006d3-dockercfg-pcq4g\"" Oct 31 19:05:21.903129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.903087 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:21.903340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.903140 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txf6\" (UniqueName: \"kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6\") pod \"payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:21.903340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.903217 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:21.903340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.903243 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hlm\" (UniqueName: \"kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm\") pod \"payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:21.903844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.903814 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:21.959654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:21.959626 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hlm\" (UniqueName: \"kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm\") pod \"payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:22.004412 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.004378 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txf6\" (UniqueName: \"kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6\") pod \"payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:22.004583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.004428 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:22.005086 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.005068 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:22.060801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.060774 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txf6\" (UniqueName: \"kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6\") pod \"payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:22.133651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.133563 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:22.210555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.210522 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:22.511539 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.511512 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn"] Oct 31 19:05:22.512961 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:05:22.512935 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3336e8a0_1ea0_49e3_a56b_7f9a0de95c22.slice/crio-7551d8d6de8657f86dcf1cdcf98edfc62307e08897e00b55ded763a91f31a043 WatchSource:0}: Error finding container 7551d8d6de8657f86dcf1cdcf98edfc62307e08897e00b55ded763a91f31a043: Status 404 returned error can't find the container with id 7551d8d6de8657f86dcf1cdcf98edfc62307e08897e00b55ded763a91f31a043 Oct 31 19:05:22.622119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:22.622096 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7"] Oct 31 19:05:22.623762 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:05:22.623736 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7761a8a_73ce_47cb_a377_4336d108c47e.slice/crio-e17e3f9b29f8fe3a4730e2ba908242b314bf3cad4ad01fedd62fbdc73973a6da WatchSource:0}: Error finding container e17e3f9b29f8fe3a4730e2ba908242b314bf3cad4ad01fedd62fbdc73973a6da: Status 404 returned error can't find the container with id e17e3f9b29f8fe3a4730e2ba908242b314bf3cad4ad01fedd62fbdc73973a6da Oct 31 19:05:23.380735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.380696 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" event={"ID":"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22","Type":"ContainerStarted","Data":"efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1"} Oct 31 19:05:23.380909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.380756 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" event={"ID":"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22","Type":"ContainerStarted","Data":"7551d8d6de8657f86dcf1cdcf98edfc62307e08897e00b55ded763a91f31a043"} Oct 31 19:05:23.380909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.380785 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:23.382533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.382495 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" event={"ID":"b7761a8a-73ce-47cb-a377-4336d108c47e","Type":"ContainerStarted","Data":"d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1"} Oct 31 19:05:23.382533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.382536 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" event={"ID":"b7761a8a-73ce-47cb-a377-4336d108c47e","Type":"ContainerStarted","Data":"e17e3f9b29f8fe3a4730e2ba908242b314bf3cad4ad01fedd62fbdc73973a6da"} Oct 31 19:05:23.382727 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.382581 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:23.496503 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:23.496454 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" podStartSLOduration=2.49644118 podStartE2EDuration="2.49644118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:23.493547241 +0000 UTC m=+2644.095688391" watchObservedRunningTime="2026-10-31 19:05:23.49644118 +0000 UTC m=+2644.098582323" Oct 31 19:05:24.084558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.084499 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" podStartSLOduration=3.084480116 podStartE2EDuration="3.084480116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:23.633834195 +0000 UTC m=+2644.235975339" watchObservedRunningTime="2026-10-31 19:05:24.084480116 +0000 UTC m=+2644.686621260" Oct 31 19:05:24.088583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.088561 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5"] Oct 31 19:05:24.092951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.092930 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145091 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145145 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145295 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145332 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145405 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5bs\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145433 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145549 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145587 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.145701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.145635 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.199128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.199095 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5"] Oct 31 19:05:24.246730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.246698 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.246890 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.246740 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.246890 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.246789 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5bs\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.246890 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.246819 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.246931 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.246972 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247020 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247073 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247121 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247145 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247400 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247418 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247481 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.247649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.247625 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.249522 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.249497 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.249661 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.249642 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.303514 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.303479 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.303662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.303555 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5bs\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.406813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.406729 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:24.838681 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.838652 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5"] Oct 31 19:05:24.840103 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:05:24.840073 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5fb9d4_3bed_4eb3_bf13_657ff9c1a16e.slice/crio-49c0624537864db5b23204361625755d645ee57f59d5677040d1e28d55d1dec0 WatchSource:0}: Error finding container 49c0624537864db5b23204361625755d645ee57f59d5677040d1e28d55d1dec0: Status 404 returned error can't find the container with id 49c0624537864db5b23204361625755d645ee57f59d5677040d1e28d55d1dec0 Oct 31 19:05:24.842813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.842777 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:05:24.842907 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.842845 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:05:24.842907 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:24.842876 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:05:24.936146 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:05:24.936090 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:05:25.397884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:25.397846 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" event={"ID":"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e","Type":"ContainerStarted","Data":"165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb"} Oct 31 19:05:25.398401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:25.397909 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" event={"ID":"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e","Type":"ContainerStarted","Data":"49c0624537864db5b23204361625755d645ee57f59d5677040d1e28d55d1dec0"} Oct 31 19:05:25.407907 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:25.407877 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:25.409213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:25.409168 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:25.409366 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:25.409240 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:25.515549 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:25.515489 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podStartSLOduration=1.5154757989999998 podStartE2EDuration="1.515475799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:25.51369472 +0000 UTC m=+2646.115835868" watchObservedRunningTime="2026-10-31 19:05:25.515475799 +0000 UTC m=+2646.117617026" Oct 31 19:05:26.407493 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:26.407460 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:26.407975 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:26.407521 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:27.407034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:27.407000 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:27.407305 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:27.407056 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:28.413862 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:28.413828 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:28.414777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:28.414740 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:29.407672 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:29.407637 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:29.407867 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:29.407701 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:30.407340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:30.407308 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:30.407767 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:30.407358 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:31.407790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:31.407755 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:05:31.408214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:31.407812 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:05:32.411702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:32.411673 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:32.412142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:32.412083 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:32.412930 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:32.412912 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:05:34.392583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:34.392549 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:05:34.392974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:05:34.392622 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:05:54.905696 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:05:54.905653 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:06:05.049391 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:05.049353 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc"] Oct 31 19:06:05.100535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:05.100501 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea0bdd-7vwnc"] Oct 31 19:06:05.566546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:05.566508 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776c0213-a96d-4d2a-850c-a3917e5bc1e9" path="/var/lib/kubelet/pods/776c0213-a96d-4d2a-850c-a3917e5bc1e9/volumes" Oct 31 19:06:19.936204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:19.936163 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:06:19.938031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:19.938009 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:06:21.044573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.044535 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp"] Oct 31 19:06:21.093624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.093594 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a453a2-d7dxp"] Oct 31 19:06:21.571675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.571639 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27371528-d4b7-402d-9a85-5cd67aa08af0" path="/var/lib/kubelet/pods/27371528-d4b7-402d-9a85-5cd67aa08af0/volumes" Oct 31 19:06:21.914014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.913936 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5"] Oct 31 19:06:21.919082 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.919061 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:21.971582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.971542 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c4731a\"" Oct 31 19:06:22.017591 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.990978 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5"] Oct 31 19:06:22.017893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:21.993318 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zkd\" (UniqueName: \"kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd\") pod \"payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.018237 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.018207 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.027484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.027452 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c4731a-dockercfg-q6xj6\"" Oct 31 19:06:22.074548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.069290 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm"] Oct 31 19:06:22.091225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.081275 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.119732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.119698 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.119904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.119818 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zkd\" (UniqueName: \"kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd\") pod \"payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.120419 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.120400 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.126373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.126342 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm"] Oct 31 19:06:22.176356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.176285 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zkd\" (UniqueName: \"kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd\") pod \"payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.221475 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.221440 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.221648 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.221606 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t989f\" (UniqueName: \"kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f\") pod \"payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.233193 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.233164 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.323309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.323269 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t989f\" (UniqueName: \"kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f\") pod \"payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.323484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.323458 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.324443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.324416 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.380165 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.380121 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t989f\" (UniqueName: \"kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f\") pod \"payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.407014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.406985 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.614663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.614634 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5"] Oct 31 19:06:22.617051 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:06:22.617025 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03045557_3751_45d5_a215_3563cd36b8a2.slice/crio-e1cb71410e925f354bad09f980610e732a5ac099c7ba944ec808a1b98d65dafc WatchSource:0}: Error finding container e1cb71410e925f354bad09f980610e732a5ac099c7ba944ec808a1b98d65dafc: Status 404 returned error can't find the container with id e1cb71410e925f354bad09f980610e732a5ac099c7ba944ec808a1b98d65dafc Oct 31 19:06:22.783204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.783159 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm"] Oct 31 19:06:22.784311 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:06:22.784284 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a88d47_278b_4f21_9831_fb4bd1359194.slice/crio-6a35f2142293d82ff641baa33ddc30b9d71d8b9fef2cafc2f7b1f63fbe5b5e48 WatchSource:0}: Error finding container 6a35f2142293d82ff641baa33ddc30b9d71d8b9fef2cafc2f7b1f63fbe5b5e48: Status 404 returned error can't find the container with id 6a35f2142293d82ff641baa33ddc30b9d71d8b9fef2cafc2f7b1f63fbe5b5e48 Oct 31 19:06:22.867707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.867665 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" event={"ID":"03045557-3751-45d5-a215-3563cd36b8a2","Type":"ContainerStarted","Data":"00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878"} Oct 31 19:06:22.867852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.867726 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" event={"ID":"03045557-3751-45d5-a215-3563cd36b8a2","Type":"ContainerStarted","Data":"e1cb71410e925f354bad09f980610e732a5ac099c7ba944ec808a1b98d65dafc"} Oct 31 19:06:22.867852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.867775 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:22.869469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.869443 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" event={"ID":"d0a88d47-278b-4f21-9831-fb4bd1359194","Type":"ContainerStarted","Data":"7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30"} Oct 31 19:06:22.869581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.869492 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" event={"ID":"d0a88d47-278b-4f21-9831-fb4bd1359194","Type":"ContainerStarted","Data":"6a35f2142293d82ff641baa33ddc30b9d71d8b9fef2cafc2f7b1f63fbe5b5e48"} Oct 31 19:06:22.869581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.869520 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:22.981165 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:22.981107 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" podStartSLOduration=1.98109267 podStartE2EDuration="1.98109267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:22.979419282 +0000 UTC m=+2703.581560425" watchObservedRunningTime="2026-10-31 19:06:22.98109267 +0000 UTC m=+2703.583233813" Oct 31 19:06:23.115301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:23.105847 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" podStartSLOduration=1.105810376 podStartE2EDuration="1.105810376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:23.09132733 +0000 UTC m=+2703.693468478" watchObservedRunningTime="2026-10-31 19:06:23.105810376 +0000 UTC m=+2703.707951520" Oct 31 19:06:24.905630 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:06:24.905592 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:06:33.878474 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:33.878403 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:06:33.878853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:33.878479 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:06:36.067529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:36.067485 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8"] Oct 31 19:06:36.097524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:36.097488 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d93f588-2p9j8"] Oct 31 19:06:37.567073 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:37.567039 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9df0e23-eb88-4ac4-8d89-744eece81be5" path="/var/lib/kubelet/pods/f9df0e23-eb88-4ac4-8d89-744eece81be5/volumes" Oct 31 19:06:40.426755 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.426719 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c"] Oct 31 19:06:40.431838 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.431818 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.487435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.487400 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c"] Oct 31 19:06:40.527586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.527549 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cl5q\" (UniqueName: \"kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.527586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.527589 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.527810 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.527754 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.533213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.533162 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:06:40.533368 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.533295 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:06:40.628778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.628742 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cl5q\" (UniqueName: \"kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.628778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.628780 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.629004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.628868 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.629490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.629469 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.631412 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.631392 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.685854 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.685783 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cl5q\" (UniqueName: \"kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q\") pod \"maas-api-revoke-sub-77b01424-lcl5c\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:40.745966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:40.745936 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:41.123175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:41.123149 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c"] Oct 31 19:06:41.125503 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:06:41.125475 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d756cfd_01a3_4b48_8925_e271fcbd157e.slice/crio-26bab7ad76783f23dfe0207911078cc380c3c9f535e1ba7d12c304eceb04d0a4 WatchSource:0}: Error finding container 26bab7ad76783f23dfe0207911078cc380c3c9f535e1ba7d12c304eceb04d0a4: Status 404 returned error can't find the container with id 26bab7ad76783f23dfe0207911078cc380c3c9f535e1ba7d12c304eceb04d0a4 Oct 31 19:06:42.052125 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:42.052098 5688 generic.go:356] "Generic (PLEG): container finished" podID="0d756cfd-01a3-4b48-8925-e271fcbd157e" containerID="334c5d1d90ee295fa079ba4c6bfba66110b6858245f99111a200b987f330f3b1" exitCode=0 Oct 31 19:06:42.052527 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:42.052206 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" event={"ID":"0d756cfd-01a3-4b48-8925-e271fcbd157e","Type":"ContainerDied","Data":"334c5d1d90ee295fa079ba4c6bfba66110b6858245f99111a200b987f330f3b1"} Oct 31 19:06:42.052527 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:42.052259 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" event={"ID":"0d756cfd-01a3-4b48-8925-e271fcbd157e","Type":"ContainerStarted","Data":"26bab7ad76783f23dfe0207911078cc380c3c9f535e1ba7d12c304eceb04d0a4"} Oct 31 19:06:43.237924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.237903 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:43.361038 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.360957 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca\") pod \"0d756cfd-01a3-4b48-8925-e271fcbd157e\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " Oct 31 19:06:43.361210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.361072 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q\" (UniqueName: \"kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q\") pod \"0d756cfd-01a3-4b48-8925-e271fcbd157e\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " Oct 31 19:06:43.361290 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.361209 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca\") pod \"0d756cfd-01a3-4b48-8925-e271fcbd157e\" (UID: \"0d756cfd-01a3-4b48-8925-e271fcbd157e\") " Oct 31 19:06:43.361528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.361500 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca" pod "0d756cfd-01a3-4b48-8925-e271fcbd157e" (UID: "0d756cfd-01a3-4b48-8925-e271fcbd157e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:06:43.361912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.361889 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:06:43.363384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.363354 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca" pod "0d756cfd-01a3-4b48-8925-e271fcbd157e" (UID: "0d756cfd-01a3-4b48-8925-e271fcbd157e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:06:43.363502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.363430 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q" pod "0d756cfd-01a3-4b48-8925-e271fcbd157e" (UID: "0d756cfd-01a3-4b48-8925-e271fcbd157e"). InnerVolumeSpecName "kube-api-access-5cl5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:06:43.463253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.463220 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d756cfd-01a3-4b48-8925-e271fcbd157e-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:06:43.463253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:43.463248 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cl5q\" (UniqueName: \"kubernetes.io/projected/0d756cfd-01a3-4b48-8925-e271fcbd157e-kube-api-access-5cl5q\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:06:44.076210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:44.075819 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" event={"ID":"0d756cfd-01a3-4b48-8925-e271fcbd157e","Type":"ContainerDied","Data":"26bab7ad76783f23dfe0207911078cc380c3c9f535e1ba7d12c304eceb04d0a4"} Oct 31 19:06:44.076210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:44.075918 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26bab7ad76783f23dfe0207911078cc380c3c9f535e1ba7d12c304eceb04d0a4" Oct 31 19:06:44.076210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:44.075990 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c" Oct 31 19:06:51.221222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.221172 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c"] Oct 31 19:06:51.221916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.221896 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0d756cfd-01a3-4b48-8925-e271fcbd157e" containerName="revoke-keys" Oct 31 19:06:51.222013 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.221918 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d756cfd-01a3-4b48-8925-e271fcbd157e" containerName="revoke-keys" Oct 31 19:06:51.222069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.222056 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d756cfd-01a3-4b48-8925-e271fcbd157e" containerName="revoke-keys" Oct 31 19:06:51.225278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.225257 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.283736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.283707 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c"] Oct 31 19:06:51.295387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.295358 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.295535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.295412 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.295617 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.295542 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zf5p\" (UniqueName: \"kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.329404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.329379 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:06:51.329548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.329450 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:06:51.396246 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.396218 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.396405 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.396297 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zf5p\" (UniqueName: \"kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.396464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.396413 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.396908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.396884 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.398848 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.398827 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.462701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.462664 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zf5p\" (UniqueName: \"kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p\") pod \"maas-api-revoke-sub-32d164a2-vpq5c\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.538563 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.538531 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:51.914471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:51.914440 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c"] Oct 31 19:06:51.916078 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:06:51.916052 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b260f4_8292_42e0_a38a_665272f72646.slice/crio-1c74a48b51847e3870274908a1b17a5f68816475027b3e59c1a87d7f2ede0ae6 WatchSource:0}: Error finding container 1c74a48b51847e3870274908a1b17a5f68816475027b3e59c1a87d7f2ede0ae6: Status 404 returned error can't find the container with id 1c74a48b51847e3870274908a1b17a5f68816475027b3e59c1a87d7f2ede0ae6 Oct 31 19:06:52.047764 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:52.047734 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk"] Oct 31 19:06:52.052330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:52.052304 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fca44f-bgkpk"] Oct 31 19:06:52.127083 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:52.127051 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" event={"ID":"39b260f4-8292-42e0-a38a-665272f72646","Type":"ContainerStarted","Data":"1c74a48b51847e3870274908a1b17a5f68816475027b3e59c1a87d7f2ede0ae6"} Oct 31 19:06:53.178205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.174520 5688 generic.go:356] "Generic (PLEG): container finished" podID="39b260f4-8292-42e0-a38a-665272f72646" containerID="221350eb162409ca8fde03061a2b56d20784994f4ffdb15554d7eb2f4f01faf0" exitCode=0 Oct 31 19:06:53.178205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.174663 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" event={"ID":"39b260f4-8292-42e0-a38a-665272f72646","Type":"ContainerDied","Data":"221350eb162409ca8fde03061a2b56d20784994f4ffdb15554d7eb2f4f01faf0"} Oct 31 19:06:53.445266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.425393 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck"] Oct 31 19:06:53.445266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.436730 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.466792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.466765 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck"] Oct 31 19:06:53.490815 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.490788 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6cfd68\"" Oct 31 19:06:53.490982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.490966 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6cfd68-dockercfg-8fksw\"" Oct 31 19:06:53.529693 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.529662 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8c82\" (UniqueName: \"kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82\") pod \"payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.529858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.529792 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.566973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.566941 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1c9eda-c202-44da-bdfd-5a16419476bf" path="/var/lib/kubelet/pods/eb1c9eda-c202-44da-bdfd-5a16419476bf/volumes" Oct 31 19:06:53.631336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.631307 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8c82\" (UniqueName: \"kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82\") pod \"payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.631506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.631406 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.632161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.632129 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.687876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.687840 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8c82\" (UniqueName: \"kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82\") pod \"payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:53.758684 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:53.758652 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:54.134033 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.134002 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck"] Oct 31 19:06:54.135726 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:06:54.135698 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884dde44_39ce_4e1b_b7e1_c59d07f01556.slice/crio-f3c7d76da35152c935e60cb1f6942c1229c123012aeae029f4503b4448f32d46 WatchSource:0}: Error finding container f3c7d76da35152c935e60cb1f6942c1229c123012aeae029f4503b4448f32d46: Status 404 returned error can't find the container with id f3c7d76da35152c935e60cb1f6942c1229c123012aeae029f4503b4448f32d46 Oct 31 19:06:54.181530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.181494 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" event={"ID":"884dde44-39ce-4e1b-b7e1-c59d07f01556","Type":"ContainerStarted","Data":"f3c7d76da35152c935e60cb1f6942c1229c123012aeae029f4503b4448f32d46"} Oct 31 19:06:54.301597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.301572 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:54.338800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.338770 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca\") pod \"39b260f4-8292-42e0-a38a-665272f72646\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " Oct 31 19:06:54.338940 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.338906 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca\") pod \"39b260f4-8292-42e0-a38a-665272f72646\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " Oct 31 19:06:54.339035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.339021 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p\" (UniqueName: \"kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p\") pod \"39b260f4-8292-42e0-a38a-665272f72646\" (UID: \"39b260f4-8292-42e0-a38a-665272f72646\") " Oct 31 19:06:54.339154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.339130 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca" pod "39b260f4-8292-42e0-a38a-665272f72646" (UID: "39b260f4-8292-42e0-a38a-665272f72646"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:06:54.339402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.339380 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b260f4-8292-42e0-a38a-665272f72646-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:06:54.341111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.341091 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca" pod "39b260f4-8292-42e0-a38a-665272f72646" (UID: "39b260f4-8292-42e0-a38a-665272f72646"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:06:54.341260 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.341123 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p" pod "39b260f4-8292-42e0-a38a-665272f72646" (UID: "39b260f4-8292-42e0-a38a-665272f72646"). InnerVolumeSpecName "kube-api-access-8zf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:06:54.441054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.440977 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b260f4-8292-42e0-a38a-665272f72646-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:06:54.441054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:54.441008 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zf5p\" (UniqueName: \"kubernetes.io/projected/39b260f4-8292-42e0-a38a-665272f72646-kube-api-access-8zf5p\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:06:54.907924 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:06:54.907887 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:06:55.188943 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:55.188912 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" Oct 31 19:06:55.188943 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:55.188933 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c" event={"ID":"39b260f4-8292-42e0-a38a-665272f72646","Type":"ContainerDied","Data":"1c74a48b51847e3870274908a1b17a5f68816475027b3e59c1a87d7f2ede0ae6"} Oct 31 19:06:55.189547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:55.188979 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c74a48b51847e3870274908a1b17a5f68816475027b3e59c1a87d7f2ede0ae6" Oct 31 19:06:55.192980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:55.192934 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" event={"ID":"884dde44-39ce-4e1b-b7e1-c59d07f01556","Type":"ContainerStarted","Data":"00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503"} Oct 31 19:06:55.193130 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:55.193008 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:06:55.305247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:06:55.305170 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" podStartSLOduration=2.305153025 podStartE2EDuration="2.305153025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:55.304444755 +0000 UTC m=+2735.906585896" watchObservedRunningTime="2026-10-31 19:06:55.305153025 +0000 UTC m=+2735.907294169" Oct 31 19:07:06.202389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:06.202359 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:07:09.049951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:09.049914 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt"] Oct 31 19:07:09.099875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:09.099843 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ce059f-tm5bt"] Oct 31 19:07:09.568134 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:09.568103 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3972e76b-9f4e-4039-a847-9caf4eb841fe" path="/var/lib/kubelet/pods/3972e76b-9f4e-4039-a847-9caf4eb841fe/volumes" Oct 31 19:07:13.347981 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.347944 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k"] Oct 31 19:07:13.348790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.348767 5688 cpu_manager.go:414] "containerMap: removing container" podUID="39b260f4-8292-42e0-a38a-665272f72646" containerName="revoke-keys" Oct 31 19:07:13.348970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.348794 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b260f4-8292-42e0-a38a-665272f72646" containerName="revoke-keys" Oct 31 19:07:13.348970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.348925 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b260f4-8292-42e0-a38a-665272f72646" containerName="revoke-keys" Oct 31 19:07:13.354215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.354167 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.456883 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.456854 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d1ff32\"" Oct 31 19:07:13.456883 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.456855 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d1ff32-dockercfg-5pw72\"" Oct 31 19:07:13.459910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.459890 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k"] Oct 31 19:07:13.501736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.501707 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.501898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.501767 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtx2j\" (UniqueName: \"kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j\") pod \"payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.603236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.603139 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.603390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.603242 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtx2j\" (UniqueName: \"kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j\") pod \"payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.603805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.603785 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.659840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.659808 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtx2j\" (UniqueName: \"kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j\") pod \"payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:13.669689 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:13.669665 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:14.075614 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.075570 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k"] Oct 31 19:07:14.077003 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:07:14.076969 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca7b0d1_f4e1_464f_90c2_11723eed536f.slice/crio-b79ff9b530dd81d911131340c6899e5e0ee6f3cd60832fffe850083b67928ef7 WatchSource:0}: Error finding container b79ff9b530dd81d911131340c6899e5e0ee6f3cd60832fffe850083b67928ef7: Status 404 returned error can't find the container with id b79ff9b530dd81d911131340c6899e5e0ee6f3cd60832fffe850083b67928ef7 Oct 31 19:07:14.360633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.360547 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" event={"ID":"fca7b0d1-f4e1-464f-90c2-11723eed536f","Type":"ContainerStarted","Data":"a1a432708b4bb4c08075040de11587406b63239bcb39609e82c4ee4c432c59ea"} Oct 31 19:07:14.360633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.360604 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" event={"ID":"fca7b0d1-f4e1-464f-90c2-11723eed536f","Type":"ContainerStarted","Data":"b79ff9b530dd81d911131340c6899e5e0ee6f3cd60832fffe850083b67928ef7"} Oct 31 19:07:14.361136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.360773 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:14.432098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.432068 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd"] Oct 31 19:07:14.436043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.436026 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.475903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.475852 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" podStartSLOduration=1.475839176 podStartE2EDuration="1.475839176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:14.473224971 +0000 UTC m=+2755.075366116" watchObservedRunningTime="2026-10-31 19:07:14.475839176 +0000 UTC m=+2755.077980314" Oct 31 19:07:14.488739 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.488714 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c4731a-serving-cert\"" Oct 31 19:07:14.492636 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.492612 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd"] Oct 31 19:07:14.518537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.518508 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.518720 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.518696 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.518787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.518733 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.518915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.518888 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdb9\" (UniqueName: \"kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.627307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.626704 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.627307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.626750 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.627307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.626809 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdb9\" (UniqueName: \"kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.627307 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:07:14.626851 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-serving-cert: secret "maas-api-e2e-shared-b-w0-c4731a-serving-cert" not found Oct 31 19:07:14.627307 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:07:14.626940 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls podName:ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e nodeName:}" failed. No retries permitted until 2026-10-31 19:07:15.126915956 +0000 UTC m=+2755.729057095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls") pod "maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" (UID: "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e") : secret "maas-api-e2e-shared-b-w0-c4731a-serving-cert" not found Oct 31 19:07:14.627307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.627018 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.629094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.627842 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.637912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.637885 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:14.683101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:14.683071 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdb9\" (UniqueName: \"kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:15.138524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:15.138482 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:15.141001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:15.140980 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:15.350261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:15.350229 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:15.731416 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:15.731387 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd"] Oct 31 19:07:15.733216 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:07:15.733160 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac03bcd2_2b9a_4a65_9ab2_cf9ec22fd19e.slice/crio-c7e1e43d939a8bbbfc58c5ab8bb8e6fda2bf48913b956406f7771ea5535a453b WatchSource:0}: Error finding container c7e1e43d939a8bbbfc58c5ab8bb8e6fda2bf48913b956406f7771ea5535a453b: Status 404 returned error can't find the container with id c7e1e43d939a8bbbfc58c5ab8bb8e6fda2bf48913b956406f7771ea5535a453b Oct 31 19:07:15.734934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:15.734912 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:07:16.376992 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:16.376957 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" event={"ID":"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e","Type":"ContainerStarted","Data":"c7e1e43d939a8bbbfc58c5ab8bb8e6fda2bf48913b956406f7771ea5535a453b"} Oct 31 19:07:17.385704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:17.385664 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" event={"ID":"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e","Type":"ContainerStarted","Data":"7b5f68936766e18efb1bc6af66470931c0528ebd0fdfef3dd94940721bacfc61"} Oct 31 19:07:17.386195 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:17.385801 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:17.498901 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:17.498851 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" podStartSLOduration=2.4151444619999998 podStartE2EDuration="3.498837425s" totalImagesPullingTime="1.083692963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:17.49829198 +0000 UTC m=+2758.100433126" watchObservedRunningTime="2026-10-31 19:07:17.498837425 +0000 UTC m=+2758.100978568" Oct 31 19:07:20.866617 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:20.866587 5688 scope.go:122] "RemoveContainer" containerID="9bc92b27dfaf901a586c9d1e02a2b7f47084472f43cf9c1bd442e82c82f51406" Oct 31 19:07:20.876002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:20.875977 5688 scope.go:122] "RemoveContainer" containerID="ef41ac41c43f6a1b65e52e0e2b19a3840a9576287ce62c43025cf0728a898373" Oct 31 19:07:23.402245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:23.402174 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:07:24.905284 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:07:24.905250 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:07:25.374461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:25.374420 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:07:31.045971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:31.045930 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx"] Oct 31 19:07:31.091670 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:31.091630 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255eadbe-89rjx"] Oct 31 19:07:31.565806 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:31.565771 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c71a842-a502-4469-bce0-6fabf68dcb65" path="/var/lib/kubelet/pods/4c71a842-a502-4469-bce0-6fabf68dcb65/volumes" Oct 31 19:07:46.048971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:46.048929 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk"] Oct 31 19:07:46.098675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:46.098639 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0e0a99-r7lnk"] Oct 31 19:07:47.565984 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:07:47.565949 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1deb11-7cd9-4fa0-9305-5d3abfd05692" path="/var/lib/kubelet/pods/9d1deb11-7cd9-4fa0-9305-5d3abfd05692/volumes" Oct 31 19:07:54.944879 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:07:54.944840 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:08:12.044208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:12.044103 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps"] Oct 31 19:08:12.047546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:12.047523 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd09fa9-mp8ps"] Oct 31 19:08:13.566056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:13.566026 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf" path="/var/lib/kubelet/pods/18ba62d2-b6e6-430d-b52a-2ee9b9d1d0cf/volumes" Oct 31 19:08:16.212500 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.212468 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w"] Oct 31 19:08:16.216398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.216376 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.265007 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.264965 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4dtb\" (UniqueName: \"kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.265178 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.265155 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.265283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.265233 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.265346 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.265325 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.319663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.319629 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-d1ff32-serving-cert\"" Oct 31 19:08:16.322626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.322605 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w"] Oct 31 19:08:16.366472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.366432 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.366657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.366507 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.366657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.366549 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.366657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.366609 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4dtb\" (UniqueName: \"kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.366835 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:08:16.366729 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-serving-cert: secret "maas-api-e2e-shared-a-w4-d1ff32-serving-cert" not found Oct 31 19:08:16.366835 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:08:16.366805 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls podName:b3ad3539-217c-40c4-a417-17577eebe8d7 nodeName:}" failed. No retries permitted until 2026-10-31 19:08:16.866784549 +0000 UTC m=+2817.468925691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls") pod "maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" (UID: "b3ad3539-217c-40c4-a417-17577eebe8d7") : secret "maas-api-e2e-shared-a-w4-d1ff32-serving-cert" not found Oct 31 19:08:16.367306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.367282 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.369141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.369118 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.423663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.423633 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4dtb\" (UniqueName: \"kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.874379 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.874346 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:16.876918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:16.876895 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:17.130296 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:17.130204 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:17.507129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:17.507103 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w"] Oct 31 19:08:17.508771 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:08:17.508744 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ad3539_217c_40c4_a417_17577eebe8d7.slice/crio-c803e2eb5680851d38ec95291032908406019c45dc37749f35b5d6d8503805d0 WatchSource:0}: Error finding container c803e2eb5680851d38ec95291032908406019c45dc37749f35b5d6d8503805d0: Status 404 returned error can't find the container with id c803e2eb5680851d38ec95291032908406019c45dc37749f35b5d6d8503805d0 Oct 31 19:08:17.809646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:17.809558 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" event={"ID":"b3ad3539-217c-40c4-a417-17577eebe8d7","Type":"ContainerStarted","Data":"c803e2eb5680851d38ec95291032908406019c45dc37749f35b5d6d8503805d0"} Oct 31 19:08:18.817367 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:18.817338 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" event={"ID":"b3ad3539-217c-40c4-a417-17577eebe8d7","Type":"ContainerStarted","Data":"88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8"} Oct 31 19:08:18.817761 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:18.817468 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:18.932276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:18.932222 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" podStartSLOduration=2.083825281 podStartE2EDuration="2.932206757s" totalImagesPullingTime="848.381476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:08:18.930024922 +0000 UTC m=+2819.532166065" watchObservedRunningTime="2026-10-31 19:08:18.932206757 +0000 UTC m=+2819.534347901" Oct 31 19:08:20.044300 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:20.044267 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq"] Oct 31 19:08:20.093719 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:20.093685 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816afeb5-b4bjq"] Oct 31 19:08:20.903840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:20.903806 5688 scope.go:122] "RemoveContainer" containerID="93f47c1c3461f60dfdb974bc190a36e7c6cf6687e61fdf63b84bab0e3c57f982" Oct 31 19:08:20.913713 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:20.913694 5688 scope.go:122] "RemoveContainer" containerID="67bd70806d3d356848fa48c66fd9e2a56497d9fe3b4a27562e83a26eff0a594f" Oct 31 19:08:20.922407 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:20.922384 5688 scope.go:122] "RemoveContainer" containerID="3c77e919e58c834554843174c9f2508fb070315403083da175e2aec67cdf977f" Oct 31 19:08:20.930804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:20.930784 5688 scope.go:122] "RemoveContainer" containerID="dfa2c6fd94472092f59656bc7bcc70ab6908da069048a43038527523091d4f12" Oct 31 19:08:21.566525 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:21.566492 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53f6490-d12c-40f5-aa31-a4ba5600bc8b" path="/var/lib/kubelet/pods/e53f6490-d12c-40f5-aa31-a4ba5600bc8b/volumes" Oct 31 19:08:24.828845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:24.828816 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:08:24.905513 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:08:24.905471 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:08:31.107843 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:31.107809 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs"] Oct 31 19:08:31.110260 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:31.110234 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7457c71-62gfs"] Oct 31 19:08:31.566461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:31.566430 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e63562-ae59-4132-81d2-4389507b540e" path="/var/lib/kubelet/pods/81e63562-ae59-4132-81d2-4389507b540e/volumes" Oct 31 19:08:37.046239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:37.046207 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7"] Oct 31 19:08:37.096089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:37.096060 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6971358f-mq2k7"] Oct 31 19:08:37.566766 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:37.566731 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8799857-4376-4dc8-b27b-1abf614320fa" path="/var/lib/kubelet/pods/f8799857-4376-4dc8-b27b-1abf614320fa/volumes" Oct 31 19:08:47.046843 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:47.046801 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9"] Oct 31 19:08:47.095684 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:47.095656 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5555b8a-klct9"] Oct 31 19:08:47.567553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:47.567520 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c8f1dd-2186-47b4-a074-5b18afcec560" path="/var/lib/kubelet/pods/83c8f1dd-2186-47b4-a074-5b18afcec560/volumes" Oct 31 19:08:53.045740 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:53.045702 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw"] Oct 31 19:08:53.095051 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:53.095021 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef2e4a-7fllw"] Oct 31 19:08:53.566372 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:08:53.566340 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6818c607-4187-4e15-a779-53bd782cdef8" path="/var/lib/kubelet/pods/6818c607-4187-4e15-a779-53bd782cdef8/volumes" Oct 31 19:08:54.911367 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:08:54.911330 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:09:08.044229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:08.044196 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c"] Oct 31 19:09:08.093954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:08.093927 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25845886-n797c"] Oct 31 19:09:09.567271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:09.567241 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1408903-0386-47d7-b9c6-ac0435eea1b9" path="/var/lib/kubelet/pods/f1408903-0386-47d7-b9c6-ac0435eea1b9/volumes" Oct 31 19:09:15.044239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.044205 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv"] Oct 31 19:09:15.093608 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.093580 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf471b15-z5xxv"] Oct 31 19:09:15.241124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.241098 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq"] Oct 31 19:09:15.241335 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.241309 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" containerID="cri-o://cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a" gracePeriod=30 Oct 31 19:09:15.247157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.247119 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw"] Oct 31 19:09:15.247373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.247329 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" containerID="cri-o://64cfce986240a0979618a39e0e765bb0fd0f95c5a2bd52d6ebff372a50d69e61" gracePeriod=30 Oct 31 19:09:15.566543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:15.566514 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726b2d22-a390-4532-b372-3f3324e71112" path="/var/lib/kubelet/pods/726b2d22-a390-4532-b372-3f3324e71112/volumes" Oct 31 19:09:20.509899 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.509839 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:09:20.553763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.553731 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:09:20.641745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.641716 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.641895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.641768 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.641895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.641832 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.641895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.641871 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.641908 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.642062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.641964 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642034 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642078 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.642267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642076 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:09:20.642267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642216 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:09:20.642267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642224 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.642267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642233 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.642267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642255 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642272 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642307 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642345 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642376 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642411 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642446 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.642560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642477 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket\") pod \"298fb7e0-12ea-4e91-8142-cb1c27880a69\" (UID: \"298fb7e0-12ea-4e91-8142-cb1c27880a69\") " Oct 31 19:09:20.642931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642566 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.642931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642596 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.642931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642602 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.642931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642610 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.642931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.642692 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket\") pod \"1f4304df-7825-46d7-81f0-83d1f789061f\" (UID: \"1f4304df-7825-46d7-81f0-83d1f789061f\") " Oct 31 19:09:20.643491 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643466 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4304df-7825-46d7-81f0-83d1f789061f-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.643580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643498 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298fb7e0-12ea-4e91-8142-cb1c27880a69-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.643580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643514 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.643580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643529 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.643580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643545 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.643580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643561 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.643580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643574 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.644089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.643997 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.644232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.644129 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.644232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.644127 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.645098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.645069 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.645214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.645147 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:09:20.645314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.645287 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:09:20.645517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.645485 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "kube-api-access-nbzcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:09:20.645627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.645516 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "kube-api-access-zkl8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:09:20.645943 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.645924 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:09:20.646138 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.646118 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy" pod "1f4304df-7825-46d7-81f0-83d1f789061f" (UID: "1f4304df-7825-46d7-81f0-83d1f789061f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:09:20.646383 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.646367 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo" pod "298fb7e0-12ea-4e91-8142-cb1c27880a69" (UID: "298fb7e0-12ea-4e91-8142-cb1c27880a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:09:20.744961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.744926 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.744961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.744950 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.744961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.744959 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.744969 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4304df-7825-46d7-81f0-83d1f789061f-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.744983 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkl8h\" (UniqueName: \"kubernetes.io/projected/298fb7e0-12ea-4e91-8142-cb1c27880a69-kube-api-access-zkl8h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.744998 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298fb7e0-12ea-4e91-8142-cb1c27880a69-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.745009 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.745016 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298fb7e0-12ea-4e91-8142-cb1c27880a69-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.745024 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.745033 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbzcv\" (UniqueName: \"kubernetes.io/projected/1f4304df-7825-46d7-81f0-83d1f789061f-kube-api-access-nbzcv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.745175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.745040 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4304df-7825-46d7-81f0-83d1f789061f-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:09:20.971975 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.971899 5688 scope.go:122] "RemoveContainer" containerID="286fdd4e3567275784323a6dbdc7ac4e60c58288a7f7e4dcf623153d78d75569" Oct 31 19:09:20.981469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.981452 5688 scope.go:122] "RemoveContainer" containerID="b0cf7abe75644b2b5c325226ac88f4761036f01186e8afc368d2e03b3e342ac6" Oct 31 19:09:20.990420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.990395 5688 scope.go:122] "RemoveContainer" containerID="64cfce986240a0979618a39e0e765bb0fd0f95c5a2bd52d6ebff372a50d69e61" Oct 31 19:09:20.999103 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:20.999079 5688 scope.go:122] "RemoveContainer" containerID="4569794f891ea51cce4ddc98dbac70144909c8721af6ee99e67c512f60c8639e" Oct 31 19:09:21.007113 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.007092 5688 scope.go:122] "RemoveContainer" containerID="3b580c5d5989320448e545584d325a26154377eaf6e4c3c8963cc27efe0aaaa6" Oct 31 19:09:21.046820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.046791 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9"] Oct 31 19:09:21.048902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.048882 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b366e594-dctl9"] Oct 31 19:09:21.281359 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281331 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f4304df-7825-46d7-81f0-83d1f789061f" containerID="cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a" exitCode=0 Oct 31 19:09:21.281513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281374 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" event={"ID":"1f4304df-7825-46d7-81f0-83d1f789061f","Type":"ContainerDied","Data":"cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a"} Oct 31 19:09:21.281513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281388 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" Oct 31 19:09:21.281513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281430 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq" event={"ID":"1f4304df-7825-46d7-81f0-83d1f789061f","Type":"ContainerDied","Data":"03b88f0af96e9dccb309cf9b7acd05f39c7952828f744540370bce954b85dc66"} Oct 31 19:09:21.281513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281456 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" event={"ID":"298fb7e0-12ea-4e91-8142-cb1c27880a69","Type":"ContainerDied","Data":"64cfce986240a0979618a39e0e765bb0fd0f95c5a2bd52d6ebff372a50d69e61"} Oct 31 19:09:21.281513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281477 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" event={"ID":"298fb7e0-12ea-4e91-8142-cb1c27880a69","Type":"ContainerDied","Data":"b43c74578275f23259c90a911cc63b7bcbe5a4292d8b9f64e850ee290f4d0556"} Oct 31 19:09:21.281513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281496 5688 scope.go:122] "RemoveContainer" containerID="cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a" Oct 31 19:09:21.281838 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.281655 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw" Oct 31 19:09:21.292111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.292092 5688 scope.go:122] "RemoveContainer" containerID="cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a" Oct 31 19:09:21.292406 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:09:21.292381 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a\": container with ID starting with cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a not found: ID does not exist" containerID="cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a" Oct 31 19:09:21.292473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.292416 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a"} err="failed to get container status \"cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a\": rpc error: code = NotFound desc = could not find container \"cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a\": container with ID starting with cdccce7b48f2fe76db701b035486cf41d5e8bf3cb87c35e38ad13d2c3d8c088a not found: ID does not exist" Oct 31 19:09:21.509405 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.509373 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw"] Oct 31 19:09:21.515875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.515809 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-fs2pw"] Oct 31 19:09:21.566786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.566763 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" path="/var/lib/kubelet/pods/298fb7e0-12ea-4e91-8142-cb1c27880a69/volumes" Oct 31 19:09:21.567158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.567145 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74abb60a-d939-4ff7-9abb-650becb0fbc0" path="/var/lib/kubelet/pods/74abb60a-d939-4ff7-9abb-650becb0fbc0/volumes" Oct 31 19:09:21.630431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.630397 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq"] Oct 31 19:09:21.630594 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:21.630456 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-t96mq"] Oct 31 19:09:23.566718 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:23.566682 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" path="/var/lib/kubelet/pods/1f4304df-7825-46d7-81f0-83d1f789061f/volumes" Oct 31 19:09:24.904520 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:09:24.904489 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:09:39.056215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:39.053511 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9"] Oct 31 19:09:39.100951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:39.100915 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae0ea62-75sl9"] Oct 31 19:09:39.567387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:39.567352 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f3f080-2c7d-4d2f-9478-55e560f03542" path="/var/lib/kubelet/pods/f6f3f080-2c7d-4d2f-9478-55e560f03542/volumes" Oct 31 19:09:46.704538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.704503 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq"] Oct 31 19:09:46.704998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.704985 5688 cpu_manager.go:414] "containerMap: removing container" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" Oct 31 19:09:46.705054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.704999 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" Oct 31 19:09:46.705054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.705037 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" Oct 31 19:09:46.705054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.705043 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" Oct 31 19:09:46.705162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.705115 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="298fb7e0-12ea-4e91-8142-cb1c27880a69" containerName="istio-proxy" Oct 31 19:09:46.705162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.705127 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4304df-7825-46d7-81f0-83d1f789061f" containerName="istio-proxy" Oct 31 19:09:46.708318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.708302 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:46.850469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.850434 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-1aefcd\"" Oct 31 19:09:46.865467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.851615 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsf9\" (UniqueName: \"kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9\") pod \"payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:46.865958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.865923 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:46.866250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.862879 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-1aefcd-dockercfg-7qjw7\"" Oct 31 19:09:46.879893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.879866 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq"] Oct 31 19:09:46.882026 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.882003 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf"] Oct 31 19:09:46.887905 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.887886 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:46.920513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.920477 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf"] Oct 31 19:09:46.967445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.967379 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsf9\" (UniqueName: \"kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9\") pod \"payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:46.967568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.967502 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:46.967568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.967561 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:46.967651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.967576 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5k7\" (UniqueName: \"kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7\") pod \"payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:46.968163 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:46.968143 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:47.023910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.023885 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsf9\" (UniqueName: \"kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9\") pod \"payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:47.068893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.068850 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:47.069014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.068902 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5k7\" (UniqueName: \"kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7\") pod \"payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:47.069578 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.069561 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:47.125389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.125365 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5k7\" (UniqueName: \"kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7\") pod \"payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:47.206740 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.206712 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:47.321306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.321279 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:47.584088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.584067 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf"] Oct 31 19:09:47.586220 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:09:47.586177 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd74f69_ecf0_419c_a821_a93c42903a38.slice/crio-8c4eb92023467370e94a6f9f1db978703318f151ef664cd8e0bfb3e484968b32 WatchSource:0}: Error finding container 8c4eb92023467370e94a6f9f1db978703318f151ef664cd8e0bfb3e484968b32: Status 404 returned error can't find the container with id 8c4eb92023467370e94a6f9f1db978703318f151ef664cd8e0bfb3e484968b32 Oct 31 19:09:47.695788 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:47.695761 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq"] Oct 31 19:09:47.698445 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:09:47.698420 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13087a8_45a8_4780_8f7a_480a745c9bd9.slice/crio-b264952863bcd957ca15bbb28aa3eb2d541c846665c87cb0e8fae0bbbdeb7e5b WatchSource:0}: Error finding container b264952863bcd957ca15bbb28aa3eb2d541c846665c87cb0e8fae0bbbdeb7e5b: Status 404 returned error can't find the container with id b264952863bcd957ca15bbb28aa3eb2d541c846665c87cb0e8fae0bbbdeb7e5b Oct 31 19:09:48.498103 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.498064 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" event={"ID":"d13087a8-45a8-4780-8f7a-480a745c9bd9","Type":"ContainerStarted","Data":"a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a"} Oct 31 19:09:48.498583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.498115 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" event={"ID":"d13087a8-45a8-4780-8f7a-480a745c9bd9","Type":"ContainerStarted","Data":"b264952863bcd957ca15bbb28aa3eb2d541c846665c87cb0e8fae0bbbdeb7e5b"} Oct 31 19:09:48.498583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.498141 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:48.499666 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.499643 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" event={"ID":"efd74f69-ecf0-419c-a821-a93c42903a38","Type":"ContainerStarted","Data":"827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8"} Oct 31 19:09:48.499763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.499684 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" event={"ID":"efd74f69-ecf0-419c-a821-a93c42903a38","Type":"ContainerStarted","Data":"8c4eb92023467370e94a6f9f1db978703318f151ef664cd8e0bfb3e484968b32"} Oct 31 19:09:48.499763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.499712 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:48.611875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:48.611811 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" podStartSLOduration=2.611793436 podStartE2EDuration="2.611793436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:09:48.610938193 +0000 UTC m=+2909.213079339" watchObservedRunningTime="2026-10-31 19:09:48.611793436 +0000 UTC m=+2909.213934580" Oct 31 19:09:54.904583 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:09:54.904544 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:09:59.508343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:59.508315 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:09:59.508746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:59.508389 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:09:59.629095 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:09:59.629047 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" podStartSLOduration=13.629033191 podStartE2EDuration="13.629033191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:09:48.730703141 +0000 UTC m=+2909.332844287" watchObservedRunningTime="2026-10-31 19:09:59.629033191 +0000 UTC m=+2920.231174336" Oct 31 19:10:10.347008 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:10.346975 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m"] Oct 31 19:10:10.347507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:10.347157 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" containerID="cri-o://80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557" gracePeriod=30 Oct 31 19:10:15.614942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.614878 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:10:15.666156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.666128 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerID="80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557" exitCode=0 Oct 31 19:10:15.666299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.666169 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" Oct 31 19:10:15.666299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.666217 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" event={"ID":"1f7f3e8b-df19-4bbd-8e94-e47bb629298d","Type":"ContainerDied","Data":"80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557"} Oct 31 19:10:15.666299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.666269 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m" event={"ID":"1f7f3e8b-df19-4bbd-8e94-e47bb629298d","Type":"ContainerDied","Data":"599f375f26a049a06acc512a995237ab80e7b2a1a3bb9b51f8fc7bba94cb2c68"} Oct 31 19:10:15.666299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.666291 5688 scope.go:122] "RemoveContainer" containerID="80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557" Oct 31 19:10:15.678163 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.678128 5688 scope.go:122] "RemoveContainer" containerID="80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557" Oct 31 19:10:15.678558 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:10:15.678539 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557\": container with ID starting with 80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557 not found: ID does not exist" containerID="80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557" Oct 31 19:10:15.678637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.678566 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557"} err="failed to get container status \"80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557\": rpc error: code = NotFound desc = could not find container \"80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557\": container with ID starting with 80e93476640f3b0690d29bdb1e2ea47b65f28028aea99cd6af90ee19d8fa8557 not found: ID does not exist" Oct 31 19:10:15.683875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.683855 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.683967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.683919 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.683967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.683951 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684014 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684075 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684061 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684090 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684172 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684128 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:15.684257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684175 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684245 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684359 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684297 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert\") pod \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\" (UID: \"1f7f3e8b-df19-4bbd-8e94-e47bb629298d\") " Oct 31 19:10:15.684419 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684362 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:15.684419 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684381 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:15.685150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684735 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:15.685150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684811 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.685150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684829 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.685150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684843 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.685150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.684906 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:10:15.686446 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.686410 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "kube-api-access-5vwbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:10:15.686446 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.686427 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:10:15.686574 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.686476 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:10:15.686820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.686795 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy" pod "1f7f3e8b-df19-4bbd-8e94-e47bb629298d" (UID: "1f7f3e8b-df19-4bbd-8e94-e47bb629298d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:15.787954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.785650 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.787954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.788006 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vwbk\" (UniqueName: \"kubernetes.io/projected/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-kube-api-access-5vwbk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.787954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.788031 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.787954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.788045 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.787954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.794391 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:15.787954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:15.794458 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f3e8b-df19-4bbd-8e94-e47bb629298d-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:16.086110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:16.086080 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m"] Oct 31 19:10:16.091507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:16.091483 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-7254m"] Oct 31 19:10:17.567881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:17.567851 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" path="/var/lib/kubelet/pods/1f7f3e8b-df19-4bbd-8e94-e47bb629298d/volumes" Oct 31 19:10:21.048088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.048053 5688 scope.go:122] "RemoveContainer" containerID="fa0460575f6c1786dae7cbe6ec2c252c25280921ef2df5f6d21032f118650a31" Oct 31 19:10:21.057443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.057423 5688 scope.go:122] "RemoveContainer" containerID="88789acfa2b8f5ee4501f1583ef80b63d83a9d21575ae5cf1e6cd0fbc3e90618" Oct 31 19:10:21.066035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.066013 5688 scope.go:122] "RemoveContainer" containerID="c08371964ec339369c42a1077188a382d818be7e1fe625bd3410078bbbbfc137" Oct 31 19:10:21.074598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.074578 5688 scope.go:122] "RemoveContainer" containerID="cccf319a521cb31cfde357f5b1adcdee7a48162b5aa19c624a6ca108ac1766cb" Oct 31 19:10:21.082823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.082804 5688 scope.go:122] "RemoveContainer" containerID="44255d57908ca7d3f8c03d52a6acd3e60a2ee3c86d99270210d5aa3d1494b5b7" Oct 31 19:10:21.090614 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.090599 5688 scope.go:122] "RemoveContainer" containerID="ab6cd37b63c04881c8c60fd313b9ae2ac65584102eea8dd29e0fad90e61eae12" Oct 31 19:10:21.098431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:21.098407 5688 scope.go:122] "RemoveContainer" containerID="4c2b251075874e3336963c34adeeddbfbd96d90cd9bdcadd7813010a13e1dc94" Oct 31 19:10:23.170064 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.170030 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r"] Oct 31 19:10:23.170756 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.170733 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" Oct 31 19:10:23.170860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.170757 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" Oct 31 19:10:23.170916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.170870 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7f3e8b-df19-4bbd-8e94-e47bb629298d" containerName="istio-proxy" Oct 31 19:10:23.176201 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.176164 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.230155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.230130 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r"] Oct 31 19:10:23.276702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.276678 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.276859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.276703 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:10:23.276859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.276713 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqvk\" (UniqueName: \"kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.276859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.276769 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:10:23.276859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.276817 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.377375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.377352 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqvk\" (UniqueName: \"kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.377536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.377398 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.377624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.377550 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.378240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.378216 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.379994 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.379973 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.433841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.433785 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqvk\" (UniqueName: \"kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk\") pod \"maas-api-revoke-sub-b9adf2df-9549r\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.488825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.488802 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:23.863698 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:23.863673 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r"] Oct 31 19:10:23.865704 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:10:23.865673 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3669f8d2_9a67_4c1c_8c68_9f4b1de8a80c.slice/crio-6923cf2599c1d678cc577a158e67e5b7f845d19c1cc4fa33ab7871b1ae2d3712 WatchSource:0}: Error finding container 6923cf2599c1d678cc577a158e67e5b7f845d19c1cc4fa33ab7871b1ae2d3712: Status 404 returned error can't find the container with id 6923cf2599c1d678cc577a158e67e5b7f845d19c1cc4fa33ab7871b1ae2d3712 Oct 31 19:10:24.721268 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:24.721235 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" event={"ID":"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c","Type":"ContainerStarted","Data":"15458262f070e277ea11a79eb51ed33d1e6a3769c84fda24abb620faa53560e8"} Oct 31 19:10:24.721640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:24.721311 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" event={"ID":"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c","Type":"ContainerStarted","Data":"6923cf2599c1d678cc577a158e67e5b7f845d19c1cc4fa33ab7871b1ae2d3712"} Oct 31 19:10:24.834487 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:24.834425 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" podStartSLOduration=1.8344074639999999 podStartE2EDuration="1.834407464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:10:24.832443998 +0000 UTC m=+2945.434585137" watchObservedRunningTime="2026-10-31 19:10:24.834407464 +0000 UTC m=+2945.436548609" Oct 31 19:10:24.962574 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:10:24.962525 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:10:25.352163 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:25.352130 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5"] Oct 31 19:10:25.352702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:25.352669 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" containerID="cri-o://165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb" gracePeriod=30 Oct 31 19:10:25.729569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:25.729488 5688 generic.go:356] "Generic (PLEG): container finished" podID="3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" containerID="15458262f070e277ea11a79eb51ed33d1e6a3769c84fda24abb620faa53560e8" exitCode=0 Oct 31 19:10:25.729932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:25.729572 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" event={"ID":"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c","Type":"ContainerDied","Data":"15458262f070e277ea11a79eb51ed33d1e6a3769c84fda24abb620faa53560e8"} Oct 31 19:10:26.878759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.878737 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:26.927526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.927497 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca\") pod \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " Oct 31 19:10:26.927659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.927570 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca\") pod \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " Oct 31 19:10:26.927725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.927661 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk\" (UniqueName: \"kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk\") pod \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\" (UID: \"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c\") " Oct 31 19:10:26.927887 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.927864 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca" pod "3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" (UID: "3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:10:26.929564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.929545 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca" pod "3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" (UID: "3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:10:26.929760 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:26.929743 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk" pod "3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" (UID: "3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c"). InnerVolumeSpecName "kube-api-access-kxqvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:10:27.029057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:27.028994 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:27.029057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:27.029020 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:27.029057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:27.029029 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxqvk\" (UniqueName: \"kubernetes.io/projected/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c-kube-api-access-kxqvk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:27.742136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:27.742106 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" Oct 31 19:10:27.742316 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:27.742143 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r" event={"ID":"3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c","Type":"ContainerDied","Data":"6923cf2599c1d678cc577a158e67e5b7f845d19c1cc4fa33ab7871b1ae2d3712"} Oct 31 19:10:27.742316 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:27.742222 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6923cf2599c1d678cc577a158e67e5b7f845d19c1cc4fa33ab7871b1ae2d3712" Oct 31 19:10:30.617554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.617146 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:10:30.667323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667295 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667329 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667355 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667408 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667480 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667509 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667590 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667627 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667655 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:30.667849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667665 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:10:30.667849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667692 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo\") pod \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\" (UID: \"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e\") " Oct 31 19:10:30.667849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667702 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:30.668037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.667842 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:30.668113 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.668090 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:30.668309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.668288 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.668395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.668315 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.668395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.668330 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.668395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.668342 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.668395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.668356 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.670531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.670486 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "kube-api-access-9c5bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:10:30.670531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.670499 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:10:30.670695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.670607 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:10:30.670695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.670679 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy" pod "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" (UID: "3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:10:30.759855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.759827 5688 generic.go:356] "Generic (PLEG): container finished" podID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerID="165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb" exitCode=0 Oct 31 19:10:30.760003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.759871 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" Oct 31 19:10:30.760003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.759919 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" event={"ID":"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e","Type":"ContainerDied","Data":"165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb"} Oct 31 19:10:30.760003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.759981 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5" event={"ID":"3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e","Type":"ContainerDied","Data":"49c0624537864db5b23204361625755d645ee57f59d5677040d1e28d55d1dec0"} Oct 31 19:10:30.760196 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.760008 5688 scope.go:122] "RemoveContainer" containerID="165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb" Oct 31 19:10:30.773415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.773390 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c5bs\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-kube-api-access-9c5bs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.773512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.773420 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.773512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.773435 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.773512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.773449 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:10:30.776309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.776285 5688 scope.go:122] "RemoveContainer" containerID="165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb" Oct 31 19:10:30.776717 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:10:30.776681 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb\": container with ID starting with 165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb not found: ID does not exist" containerID="165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb" Oct 31 19:10:30.776818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.776721 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb"} err="failed to get container status \"165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb\": rpc error: code = NotFound desc = could not find container \"165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb\": container with ID starting with 165ee5f102a716d1e3ee444153858dc1f3f5f014b4423021662545f6e9da07bb not found: ID does not exist" Oct 31 19:10:30.911092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.911001 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5"] Oct 31 19:10:30.913673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:30.913646 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c22w5"] Oct 31 19:10:31.566661 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:31.566631 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" path="/var/lib/kubelet/pods/3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e/volumes" Oct 31 19:10:48.794875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.794838 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm"] Oct 31 19:10:48.795402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.795374 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" containerName="revoke-keys" Oct 31 19:10:48.795402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.795387 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" containerName="revoke-keys" Oct 31 19:10:48.795482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.795411 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" Oct 31 19:10:48.795482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.795416 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" Oct 31 19:10:48.795543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.795500 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" containerName="revoke-keys" Oct 31 19:10:48.795543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.795512 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5fb9d4-3bed-4eb3-bf13-657ff9c1a16e" containerName="istio-proxy" Oct 31 19:10:48.798631 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.798615 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:48.853398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.853373 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-1aefcd-serving-cert\"" Oct 31 19:10:48.904808 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.904772 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm"] Oct 31 19:10:48.955776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.955748 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:48.955921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.955789 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:48.955921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.955903 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkh6m\" (UniqueName: \"kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:48.955997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:48.955942 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.069352 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.069257 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.077361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.074166 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.077361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.074371 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkh6m\" (UniqueName: \"kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.077361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.074438 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.077361 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:10:49.074663 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-serving-cert: secret "maas-api-e2e-shared-b-w4-1aefcd-serving-cert" not found Oct 31 19:10:49.077361 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:10:49.074732 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls podName:6ff64189-a47e-41b0-9438-fad7681635b6 nodeName:}" failed. No retries permitted until 2026-10-31 19:10:49.574709026 +0000 UTC m=+2970.176850150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls") pod "maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" (UID: "6ff64189-a47e-41b0-9438-fad7681635b6") : secret "maas-api-e2e-shared-b-w4-1aefcd-serving-cert" not found Oct 31 19:10:49.077361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.075750 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.087829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.086688 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.134703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.134670 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkh6m\" (UniqueName: \"kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.608517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.608482 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.610976 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.610957 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:49.711827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:49.711804 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:50.085711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:50.085685 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm"] Oct 31 19:10:50.088443 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:10:50.088416 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff64189_a47e_41b0_9438_fad7681635b6.slice/crio-86b2152c3b798ef8193b047ec7479119fa70bdf82669e8dec5f70e79c56e1ec9 WatchSource:0}: Error finding container 86b2152c3b798ef8193b047ec7479119fa70bdf82669e8dec5f70e79c56e1ec9: Status 404 returned error can't find the container with id 86b2152c3b798ef8193b047ec7479119fa70bdf82669e8dec5f70e79c56e1ec9 Oct 31 19:10:50.893456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:50.893415 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" event={"ID":"6ff64189-a47e-41b0-9438-fad7681635b6","Type":"ContainerStarted","Data":"86b2152c3b798ef8193b047ec7479119fa70bdf82669e8dec5f70e79c56e1ec9"} Oct 31 19:10:51.900349 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:51.900314 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" event={"ID":"6ff64189-a47e-41b0-9438-fad7681635b6","Type":"ContainerStarted","Data":"0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606"} Oct 31 19:10:51.900720 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:51.900382 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:10:52.015632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:52.015589 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" podStartSLOduration=3.129290018 podStartE2EDuration="4.015575144s" totalImagesPullingTime="886.285126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:10:52.013802032 +0000 UTC m=+2972.615943176" watchObservedRunningTime="2026-10-31 19:10:52.015575144 +0000 UTC m=+2972.617716288" Oct 31 19:10:54.904376 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:10:54.904328 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:10:57.911745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:10:57.911667 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:11:11.238850 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.238817 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j"] Oct 31 19:11:11.243344 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.243321 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.297609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.297574 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:11:11.318269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.318242 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.318428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.318376 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.318531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.318513 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89bv\" (UniqueName: \"kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.347064 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.347044 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:11:11.349383 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.349363 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j"] Oct 31 19:11:11.419962 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.419938 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.420081 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.420006 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.420152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.420120 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89bv\" (UniqueName: \"kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.420691 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.420630 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.422473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.422454 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.477081 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.477051 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89bv\" (UniqueName: \"kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv\") pod \"maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.556978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.556901 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:11.931701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:11.931673 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j"] Oct 31 19:11:11.934616 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:11:11.934585 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64c0925_9808_4bc9_a5fc_c776e1e63dd3.slice/crio-d9c145350e9bfb2f34479ecf88fcb8bc278feeec9452f44e4289f7b4ebc11192 WatchSource:0}: Error finding container d9c145350e9bfb2f34479ecf88fcb8bc278feeec9452f44e4289f7b4ebc11192: Status 404 returned error can't find the container with id d9c145350e9bfb2f34479ecf88fcb8bc278feeec9452f44e4289f7b4ebc11192 Oct 31 19:11:12.037071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:12.037043 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" event={"ID":"e64c0925-9808-4bc9-a5fc-c776e1e63dd3","Type":"ContainerStarted","Data":"d9c145350e9bfb2f34479ecf88fcb8bc278feeec9452f44e4289f7b4ebc11192"} Oct 31 19:11:13.045816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:13.045779 5688 generic.go:356] "Generic (PLEG): container finished" podID="e64c0925-9808-4bc9-a5fc-c776e1e63dd3" containerID="433d37be898dc6ee031e98b6adf134e046a8854bff98b5cf2363a73e1b9638fe" exitCode=0 Oct 31 19:11:13.046241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:13.045858 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" event={"ID":"e64c0925-9808-4bc9-a5fc-c776e1e63dd3","Type":"ContainerDied","Data":"433d37be898dc6ee031e98b6adf134e046a8854bff98b5cf2363a73e1b9638fe"} Oct 31 19:11:14.229114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.229085 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:14.358078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.358005 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv\" (UniqueName: \"kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv\") pod \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " Oct 31 19:11:14.358267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.358145 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca\") pod \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " Oct 31 19:11:14.358267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.358169 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca\") pod \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\" (UID: \"e64c0925-9808-4bc9-a5fc-c776e1e63dd3\") " Oct 31 19:11:14.358536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.358512 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca" pod "e64c0925-9808-4bc9-a5fc-c776e1e63dd3" (UID: "e64c0925-9808-4bc9-a5fc-c776e1e63dd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:11:14.358622 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.358609 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:14.360366 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.360337 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca" pod "e64c0925-9808-4bc9-a5fc-c776e1e63dd3" (UID: "e64c0925-9808-4bc9-a5fc-c776e1e63dd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:11:14.360473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.360387 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv" pod "e64c0925-9808-4bc9-a5fc-c776e1e63dd3" (UID: "e64c0925-9808-4bc9-a5fc-c776e1e63dd3"). InnerVolumeSpecName "kube-api-access-h89bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:11:14.459170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.459143 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:14.459290 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:14.459171 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h89bv\" (UniqueName: \"kubernetes.io/projected/e64c0925-9808-4bc9-a5fc-c776e1e63dd3-kube-api-access-h89bv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:15.057717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:15.057688 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" Oct 31 19:11:15.057876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:15.057713 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j" event={"ID":"e64c0925-9808-4bc9-a5fc-c776e1e63dd3","Type":"ContainerDied","Data":"d9c145350e9bfb2f34479ecf88fcb8bc278feeec9452f44e4289f7b4ebc11192"} Oct 31 19:11:15.057876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:15.057790 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c145350e9bfb2f34479ecf88fcb8bc278feeec9452f44e4289f7b4ebc11192" Oct 31 19:11:19.973561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:19.973534 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:11:19.977546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:19.977519 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:11:21.152748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:21.152717 5688 scope.go:122] "RemoveContainer" containerID="1b4cee4a885537c7a23a1bcd8e414374080f943089329a55de50df5b214acbef" Oct 31 19:11:24.905123 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:11:24.905084 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:11:37.308792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:37.308761 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j"] Oct 31 19:11:37.358278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:37.358246 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a2c587-nwn7j"] Oct 31 19:11:37.566289 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:37.566177 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64c0925-9808-4bc9-a5fc-c776e1e63dd3" path="/var/lib/kubelet/pods/e64c0925-9808-4bc9-a5fc-c776e1e63dd3/volumes" Oct 31 19:11:44.044579 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:44.044547 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c"] Oct 31 19:11:44.094097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:44.094065 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b01424-lcl5c"] Oct 31 19:11:45.157624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:45.157593 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn"] Oct 31 19:11:45.265951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:45.265822 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7"] Oct 31 19:11:45.566988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:45.566954 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d756cfd-01a3-4b48-8925-e271fcbd157e" path="/var/lib/kubelet/pods/0d756cfd-01a3-4b48-8925-e271fcbd157e/volumes" Oct 31 19:11:50.158661 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.158618 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" podUID="3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" containerName="payload-processing" containerID="cri-o://efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1" gracePeriod=25 Oct 31 19:11:50.266398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.266363 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" podUID="b7761a8a-73ce-47cb-a377-4336d108c47e" containerName="payload-pre-processing" containerID="cri-o://d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1" gracePeriod=25 Oct 31 19:11:50.421470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.421412 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:11:50.532652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.532627 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:11:50.556512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.556480 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume\") pod \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " Oct 31 19:11:50.556690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.556615 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6\" (UniqueName: \"kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6\") pod \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\" (UID: \"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22\") " Oct 31 19:11:50.556894 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.556862 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume" pod "3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" (UID: "3336e8a0-1ea0-49e3-a56b-7f9a0de95c22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:11:50.557090 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.557072 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:50.558818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.558793 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6" pod "3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" (UID: "3336e8a0-1ea0-49e3-a56b-7f9a0de95c22"). InnerVolumeSpecName "kube-api-access-9txf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:11:50.658100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.658068 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm\" (UniqueName: \"kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm\") pod \"b7761a8a-73ce-47cb-a377-4336d108c47e\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " Oct 31 19:11:50.658293 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.658136 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume\") pod \"b7761a8a-73ce-47cb-a377-4336d108c47e\" (UID: \"b7761a8a-73ce-47cb-a377-4336d108c47e\") " Oct 31 19:11:50.658582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.658552 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume" pod "b7761a8a-73ce-47cb-a377-4336d108c47e" (UID: "b7761a8a-73ce-47cb-a377-4336d108c47e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:11:50.658716 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.658618 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txf6\" (UniqueName: \"kubernetes.io/projected/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22-kube-api-access-9txf6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:50.660699 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.660673 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm" pod "b7761a8a-73ce-47cb-a377-4336d108c47e" (UID: "b7761a8a-73ce-47cb-a377-4336d108c47e"). InnerVolumeSpecName "kube-api-access-77hlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:11:50.759482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.759452 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77hlm\" (UniqueName: \"kubernetes.io/projected/b7761a8a-73ce-47cb-a377-4336d108c47e-kube-api-access-77hlm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:50.759482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:50.759475 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7761a8a-73ce-47cb-a377-4336d108c47e-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:11:51.261242 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.261171 5688 generic.go:356] "Generic (PLEG): container finished" podID="3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" containerID="efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1" exitCode=0 Oct 31 19:11:51.261705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.261248 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" Oct 31 19:11:51.261705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.261239 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" event={"ID":"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22","Type":"ContainerDied","Data":"efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1"} Oct 31 19:11:51.261705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.261402 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn" event={"ID":"3336e8a0-1ea0-49e3-a56b-7f9a0de95c22","Type":"ContainerDied","Data":"7551d8d6de8657f86dcf1cdcf98edfc62307e08897e00b55ded763a91f31a043"} Oct 31 19:11:51.261705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.261440 5688 scope.go:122] "RemoveContainer" containerID="efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1" Oct 31 19:11:51.284898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.263828 5688 generic.go:356] "Generic (PLEG): container finished" podID="b7761a8a-73ce-47cb-a377-4336d108c47e" containerID="d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1" exitCode=0 Oct 31 19:11:51.284898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.263933 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" event={"ID":"b7761a8a-73ce-47cb-a377-4336d108c47e","Type":"ContainerDied","Data":"d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1"} Oct 31 19:11:51.284898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.263981 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" event={"ID":"b7761a8a-73ce-47cb-a377-4336d108c47e","Type":"ContainerDied","Data":"e17e3f9b29f8fe3a4730e2ba908242b314bf3cad4ad01fedd62fbdc73973a6da"} Oct 31 19:11:51.284898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.264059 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7" Oct 31 19:11:51.303043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.303018 5688 scope.go:122] "RemoveContainer" containerID="efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1" Oct 31 19:11:51.321052 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:11:51.321011 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1\": container with ID starting with efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1 not found: ID does not exist" containerID="efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1" Oct 31 19:11:51.321240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.321055 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1"} err="failed to get container status \"efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1\": rpc error: code = NotFound desc = could not find container \"efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1\": container with ID starting with efd62b3fd51f606257691ffadfc107a8e1e0215deb8663d21049e755248514c1 not found: ID does not exist" Oct 31 19:11:51.321240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.321075 5688 scope.go:122] "RemoveContainer" containerID="d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1" Oct 31 19:11:51.331001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.330978 5688 scope.go:122] "RemoveContainer" containerID="d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1" Oct 31 19:11:51.331321 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:11:51.331298 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1\": container with ID starting with d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1 not found: ID does not exist" containerID="d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1" Oct 31 19:11:51.331417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.331338 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1"} err="failed to get container status \"d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1\": rpc error: code = NotFound desc = could not find container \"d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1\": container with ID starting with d89574b35f09dad565937d09ec840b8f948f628540215769ce82efa991fb12a1 not found: ID does not exist" Oct 31 19:11:51.425152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.425123 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7"] Oct 31 19:11:51.474707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.474647 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7006d3-86f6b6ff55-qddx7"] Oct 31 19:11:51.565918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.565887 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7761a8a-73ce-47cb-a377-4336d108c47e" path="/var/lib/kubelet/pods/b7761a8a-73ce-47cb-a377-4336d108c47e/volumes" Oct 31 19:11:51.582235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.582203 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn"] Oct 31 19:11:51.585862 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:51.585840 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7006d3-c8c465dcc-72vwn"] Oct 31 19:11:53.565443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:53.565404 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" path="/var/lib/kubelet/pods/3336e8a0-1ea0-49e3-a56b-7f9a0de95c22/volumes" Oct 31 19:11:54.904551 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:11:54.904518 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:11:55.044353 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:55.044321 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c"] Oct 31 19:11:55.080533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:55.080506 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d164a2-vpq5c"] Oct 31 19:11:55.565903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:11:55.565873 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b260f4-8292-42e0-a38a-665272f72646" path="/var/lib/kubelet/pods/39b260f4-8292-42e0-a38a-665272f72646/volumes" Oct 31 19:12:24.903913 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:12:24.903877 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:12:46.776664 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:46.776632 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp"] Oct 31 19:12:46.777091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:46.776835 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" containerID="cri-o://6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c" gracePeriod=30 Oct 31 19:12:51.379877 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:51.379843 5688 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:12:51.382476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:51.379901 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:12:52.030194 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.030157 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:12:52.127263 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127170 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127414 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127291 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127414 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127348 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127414 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127405 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127579 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127435 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127641 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127618 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127692 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127661 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:12:52.127752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127694 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.127809 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127774 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:12:52.127910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127878 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:12:52.128033 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127924 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.128033 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.127980 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo\") pod \"8c031e57-604b-42b5-8896-7798e48f749c\" (UID: \"8c031e57-604b-42b5-8896-7798e48f749c\") " Oct 31 19:12:52.128328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128235 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:12:52.128401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128195 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:12:52.128547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128516 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c031e57-604b-42b5-8896-7798e48f749c-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.128547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128545 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.128875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128561 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.128875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128576 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.128875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.128590 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.129950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.129926 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:12:52.130197 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.130156 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "kube-api-access-zbm4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:12:52.130318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.130298 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:12:52.130378 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.130335 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy" pod "8c031e57-604b-42b5-8896-7798e48f749c" (UID: "8c031e57-604b-42b5-8896-7798e48f749c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:12:52.229620 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.229594 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c031e57-604b-42b5-8896-7798e48f749c-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.229620 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.229619 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbm4m\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-kube-api-access-zbm4m\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.229790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.229628 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c031e57-604b-42b5-8896-7798e48f749c-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.229790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.229638 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c031e57-604b-42b5-8896-7798e48f749c-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:12:52.617859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.617822 5688 generic.go:356] "Generic (PLEG): container finished" podID="8c031e57-604b-42b5-8896-7798e48f749c" containerID="6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c" exitCode=0 Oct 31 19:12:52.618328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.617878 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" Oct 31 19:12:52.618328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.617909 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" event={"ID":"8c031e57-604b-42b5-8896-7798e48f749c","Type":"ContainerDied","Data":"6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c"} Oct 31 19:12:52.618328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.617960 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp" event={"ID":"8c031e57-604b-42b5-8896-7798e48f749c","Type":"ContainerDied","Data":"4dd606056991ca4ccdb6d468850d6b26b1bfdd6199c045a8b700ebfb01e8f4a1"} Oct 31 19:12:52.618328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.617988 5688 scope.go:122] "RemoveContainer" containerID="6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c" Oct 31 19:12:52.628562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.628544 5688 scope.go:122] "RemoveContainer" containerID="6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c" Oct 31 19:12:52.628804 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:12:52.628784 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c\": container with ID starting with 6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c not found: ID does not exist" containerID="6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c" Oct 31 19:12:52.628858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.628812 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c"} err="failed to get container status \"6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c\": rpc error: code = NotFound desc = could not find container \"6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c\": container with ID starting with 6889101995e5c2194631a9d5ffc206f7a057f54f6dea650865bd364d4a46b20c not found: ID does not exist" Oct 31 19:12:52.781013 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.780983 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp"] Oct 31 19:12:52.830632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:52.830599 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-x8bhp"] Oct 31 19:12:53.566164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:12:53.566131 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c031e57-604b-42b5-8896-7798e48f749c" path="/var/lib/kubelet/pods/8c031e57-604b-42b5-8896-7798e48f749c/volumes" Oct 31 19:12:54.904115 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:12:54.904078 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:13:21.209065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:21.209036 5688 scope.go:122] "RemoveContainer" containerID="334c5d1d90ee295fa079ba4c6bfba66110b6858245f99111a200b987f330f3b1" Oct 31 19:13:21.218057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:21.218035 5688 scope.go:122] "RemoveContainer" containerID="221350eb162409ca8fde03061a2b56d20784994f4ffdb15554d7eb2f4f01faf0" Oct 31 19:13:24.904492 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:13:24.904455 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:13:38.746250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746212 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl"] Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746680 5688 cpu_manager.go:414] "containerMap: removing container" podUID="b7761a8a-73ce-47cb-a377-4336d108c47e" containerName="payload-pre-processing" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746697 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7761a8a-73ce-47cb-a377-4336d108c47e" containerName="payload-pre-processing" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746735 5688 cpu_manager.go:414] "containerMap: removing container" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746742 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746756 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e64c0925-9808-4bc9-a5fc-c776e1e63dd3" containerName="revoke-keys" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746764 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64c0925-9808-4bc9-a5fc-c776e1e63dd3" containerName="revoke-keys" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746785 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" containerName="payload-processing" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746791 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" containerName="payload-processing" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746917 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c031e57-604b-42b5-8896-7798e48f749c" containerName="istio-proxy" Oct 31 19:13:38.746916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746925 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64c0925-9808-4bc9-a5fc-c776e1e63dd3" containerName="revoke-keys" Oct 31 19:13:38.747338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746933 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7761a8a-73ce-47cb-a377-4336d108c47e" containerName="payload-pre-processing" Oct 31 19:13:38.747338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.746940 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3336e8a0-1ea0-49e3-a56b-7f9a0de95c22" containerName="payload-processing" Oct 31 19:13:38.751567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.751553 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.805434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.805351 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"kube-root-ca.crt\"" Oct 31 19:13:38.853898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.853872 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"openshift-service-ca.crt\"" Oct 31 19:13:38.854126 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.854106 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"default-dockercfg-5c4w4\"" Oct 31 19:13:38.854228 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.854144 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"auth-test-model-1aefcd-kserve-self-signed-certs\"" Oct 31 19:13:38.856176 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.856157 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl"] Oct 31 19:13:38.888323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.888290 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trxm2\" (UniqueName: \"kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.889633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.889597 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.889906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.889884 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.890130 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.890114 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.890631 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.890266 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.890631 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.890311 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.991754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.991712 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trxm2\" (UniqueName: \"kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.991919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.991802 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.991919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.991876 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.992009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.991987 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.992067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.992030 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.992111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.992066 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.992325 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.992297 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.992464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.992339 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.992464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.992426 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.994422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.994400 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:38.994809 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:38.994783 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:39.048871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:39.048801 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trxm2\" (UniqueName: \"kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2\") pod \"auth-test-model-1aefcd-kserve-6c775946b9-m7ssl\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:39.063546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:39.063521 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:39.446811 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:39.446720 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl"] Oct 31 19:13:39.449265 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:13:39.449238 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81ed5b92_f253_47d7_98be_ffb3d4fdfcdc.slice/crio-cd20a2c9f2a23532c202cba93066efc07e1d34ad19a7ebd7110530d3d663919f WatchSource:0}: Error finding container cd20a2c9f2a23532c202cba93066efc07e1d34ad19a7ebd7110530d3d663919f: Status 404 returned error can't find the container with id cd20a2c9f2a23532c202cba93066efc07e1d34ad19a7ebd7110530d3d663919f Oct 31 19:13:39.876741 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:39.876710 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" event={"ID":"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc","Type":"ContainerStarted","Data":"4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c"} Oct 31 19:13:39.877140 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:39.876748 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" event={"ID":"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc","Type":"ContainerStarted","Data":"cd20a2c9f2a23532c202cba93066efc07e1d34ad19a7ebd7110530d3d663919f"} Oct 31 19:13:39.990941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:39.990895 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" podStartSLOduration=1.990882619 podStartE2EDuration="1.990882619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:13:39.989548661 +0000 UTC m=+3140.591689812" watchObservedRunningTime="2026-10-31 19:13:39.990882619 +0000 UTC m=+3140.593023762" Oct 31 19:13:40.064561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:40.064531 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:47.275786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:47.275753 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g"] Oct 31 19:13:47.276319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:47.276057 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" containerID="cri-o://45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee" gracePeriod=30 Oct 31 19:13:49.064468 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:49.064434 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:49.076401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:49.076378 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:49.944753 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:49.944720 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:13:50.710891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:50.710859 5688 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:13:50.711302 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:50.710925 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:13:52.531751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.531725 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:13:52.636210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636116 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636161 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636204 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636233 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636268 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636293 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636322 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636356 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636429 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket\") pod \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\" (UID: \"c9f697de-be92-4ec9-bd0b-05fce7f411f8\") " Oct 31 19:13:52.636472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636459 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:13:52.636827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636470 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:13:52.636827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636702 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:13:52.636827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636781 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:13:52.636827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636788 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:13:52.637032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636918 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.637032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636938 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.637032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636952 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.637032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636966 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.637032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.636979 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.638633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.638610 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "kube-api-access-q955s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:13:52.638840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.638816 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:13:52.639068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.639048 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:13:52.639068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.639061 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo" pod "c9f697de-be92-4ec9-bd0b-05fce7f411f8" (UID: "c9f697de-be92-4ec9-bd0b-05fce7f411f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:13:52.738456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.738427 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q955s\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-kube-api-access-q955s\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.738456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.738451 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.738456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.738460 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.738672 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.738469 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f697de-be92-4ec9-bd0b-05fce7f411f8-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:13:52.951961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.951892 5688 generic.go:356] "Generic (PLEG): container finished" podID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerID="45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee" exitCode=0 Oct 31 19:13:52.951961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.951938 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" Oct 31 19:13:52.952166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.951980 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" event={"ID":"c9f697de-be92-4ec9-bd0b-05fce7f411f8","Type":"ContainerDied","Data":"45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee"} Oct 31 19:13:52.952166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.952036 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g" event={"ID":"c9f697de-be92-4ec9-bd0b-05fce7f411f8","Type":"ContainerDied","Data":"732a9b2a86d12280adea2e0f1d6a75f28661d6068986a6221b0d8c80f8dbf433"} Oct 31 19:13:52.952166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.952066 5688 scope.go:122] "RemoveContainer" containerID="45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee" Oct 31 19:13:52.962763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.962747 5688 scope.go:122] "RemoveContainer" containerID="45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee" Oct 31 19:13:52.963018 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:13:52.962999 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee\": container with ID starting with 45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee not found: ID does not exist" containerID="45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee" Oct 31 19:13:52.963067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:52.963027 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee"} err="failed to get container status \"45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee\": rpc error: code = NotFound desc = could not find container \"45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee\": container with ID starting with 45a3c98c7b4f802ecc179069e74d05021496bece1c227db9bf21be088b4835ee not found: ID does not exist" Oct 31 19:13:53.116857 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:53.116833 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g"] Oct 31 19:13:53.121276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:53.121254 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-x6t2g"] Oct 31 19:13:53.565665 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:13:53.565635 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" path="/var/lib/kubelet/pods/c9f697de-be92-4ec9-bd0b-05fce7f411f8/volumes" Oct 31 19:13:54.911455 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:13:54.911420 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:14:24.904821 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:14:24.904788 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:14:25.685337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.685306 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg"] Oct 31 19:14:25.685759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.685746 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" Oct 31 19:14:25.685808 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.685760 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" Oct 31 19:14:25.685857 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.685848 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f697de-be92-4ec9-bd0b-05fce7f411f8" containerName="istio-proxy" Oct 31 19:14:25.690084 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.690070 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.745551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.745524 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg"] Oct 31 19:14:25.792327 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.792292 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:14:25.792484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.792454 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:14:25.858919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.858887 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.859075 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.859060 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.859141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.859091 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrmx\" (UniqueName: \"kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.959831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.959772 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.959831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.959807 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrmx\" (UniqueName: \"kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.960223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.959839 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.960542 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.960526 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:25.962329 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:25.962307 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:26.016894 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:26.016865 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrmx\" (UniqueName: \"kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx\") pod \"maas-api-revoke-sub-22173abe-4t5wg\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:26.304011 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:26.303984 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:26.683375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:26.683297 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg"] Oct 31 19:14:26.687115 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:14:26.687087 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b12386_78be_41f8_a3a4_13e355da0ef8.slice/crio-afc8973616be3db40e818fa4c1e1bc16c7fa3ed596632fa30f9960284d78516d WatchSource:0}: Error finding container afc8973616be3db40e818fa4c1e1bc16c7fa3ed596632fa30f9960284d78516d: Status 404 returned error can't find the container with id afc8973616be3db40e818fa4c1e1bc16c7fa3ed596632fa30f9960284d78516d Oct 31 19:14:27.133223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:27.133176 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" event={"ID":"51b12386-78be-41f8-a3a4-13e355da0ef8","Type":"ContainerStarted","Data":"a14d6b1a1d0e5cf31ea1db13ca06cb2a623638b5a7a953016bb067751f0ddbcd"} Oct 31 19:14:27.133590 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:27.133253 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" event={"ID":"51b12386-78be-41f8-a3a4-13e355da0ef8","Type":"ContainerStarted","Data":"afc8973616be3db40e818fa4c1e1bc16c7fa3ed596632fa30f9960284d78516d"} Oct 31 19:14:27.247198 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:27.247129 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" podStartSLOduration=2.247116656 podStartE2EDuration="2.247116656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:14:27.245105287 +0000 UTC m=+3187.847246430" watchObservedRunningTime="2026-10-31 19:14:27.247116656 +0000 UTC m=+3187.849257799" Oct 31 19:14:28.140203 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:28.140147 5688 generic.go:356] "Generic (PLEG): container finished" podID="51b12386-78be-41f8-a3a4-13e355da0ef8" containerID="a14d6b1a1d0e5cf31ea1db13ca06cb2a623638b5a7a953016bb067751f0ddbcd" exitCode=0 Oct 31 19:14:28.140630 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:28.140236 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" event={"ID":"51b12386-78be-41f8-a3a4-13e355da0ef8","Type":"ContainerDied","Data":"a14d6b1a1d0e5cf31ea1db13ca06cb2a623638b5a7a953016bb067751f0ddbcd"} Oct 31 19:14:29.275836 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.275812 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:29.396240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.396160 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca\") pod \"51b12386-78be-41f8-a3a4-13e355da0ef8\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " Oct 31 19:14:29.396415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.396396 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca\") pod \"51b12386-78be-41f8-a3a4-13e355da0ef8\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " Oct 31 19:14:29.396478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.396434 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx\" (UniqueName: \"kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx\") pod \"51b12386-78be-41f8-a3a4-13e355da0ef8\" (UID: \"51b12386-78be-41f8-a3a4-13e355da0ef8\") " Oct 31 19:14:29.396680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.396659 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca" pod "51b12386-78be-41f8-a3a4-13e355da0ef8" (UID: "51b12386-78be-41f8-a3a4-13e355da0ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:14:29.396792 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.396771 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:14:29.398546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.398520 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca" pod "51b12386-78be-41f8-a3a4-13e355da0ef8" (UID: "51b12386-78be-41f8-a3a4-13e355da0ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:14:29.398934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.398912 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx" pod "51b12386-78be-41f8-a3a4-13e355da0ef8" (UID: "51b12386-78be-41f8-a3a4-13e355da0ef8"). InnerVolumeSpecName "kube-api-access-xhrmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:14:29.497930 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.497902 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhrmx\" (UniqueName: \"kubernetes.io/projected/51b12386-78be-41f8-a3a4-13e355da0ef8-kube-api-access-xhrmx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:14:29.497930 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:29.497928 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b12386-78be-41f8-a3a4-13e355da0ef8-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:14:30.151362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:30.151327 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" event={"ID":"51b12386-78be-41f8-a3a4-13e355da0ef8","Type":"ContainerDied","Data":"afc8973616be3db40e818fa4c1e1bc16c7fa3ed596632fa30f9960284d78516d"} Oct 31 19:14:30.151362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:30.151355 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg" Oct 31 19:14:30.151555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:30.151376 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc8973616be3db40e818fa4c1e1bc16c7fa3ed596632fa30f9960284d78516d" Oct 31 19:14:36.415282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.415248 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2"] Oct 31 19:14:36.415748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.415735 5688 cpu_manager.go:414] "containerMap: removing container" podUID="51b12386-78be-41f8-a3a4-13e355da0ef8" containerName="revoke-keys" Oct 31 19:14:36.415795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.415748 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b12386-78be-41f8-a3a4-13e355da0ef8" containerName="revoke-keys" Oct 31 19:14:36.415829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.415819 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b12386-78be-41f8-a3a4-13e355da0ef8" containerName="revoke-keys" Oct 31 19:14:36.419218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.419173 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.459606 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.459580 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.459720 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.459617 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjbx\" (UniqueName: \"kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.459766 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.459726 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.476408 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.476384 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2"] Oct 31 19:14:36.523456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.523433 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:14:36.523568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.523468 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:14:36.560440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.560410 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.560573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.560464 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjbx\" (UniqueName: \"kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.560573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.560550 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.561096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.561072 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.562994 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.562978 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.616651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.616623 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjbx\" (UniqueName: \"kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx\") pod \"maas-api-revoke-sub-860ea8a9-fd7s2\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:36.731981 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:36.731925 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:37.105371 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:37.105348 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2"] Oct 31 19:14:37.107780 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:14:37.107754 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe95fb3_e815_42fe_a591_aeee67dce92f.slice/crio-03f46e3f6d73a2ba4293a00018bdaa0cd53acb218f4ccf00f3b1b3c625f8ddfe WatchSource:0}: Error finding container 03f46e3f6d73a2ba4293a00018bdaa0cd53acb218f4ccf00f3b1b3c625f8ddfe: Status 404 returned error can't find the container with id 03f46e3f6d73a2ba4293a00018bdaa0cd53acb218f4ccf00f3b1b3c625f8ddfe Oct 31 19:14:37.190404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:37.190250 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" event={"ID":"ebe95fb3-e815-42fe-a591-aeee67dce92f","Type":"ContainerStarted","Data":"03f46e3f6d73a2ba4293a00018bdaa0cd53acb218f4ccf00f3b1b3c625f8ddfe"} Oct 31 19:14:38.198336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:38.198297 5688 generic.go:356] "Generic (PLEG): container finished" podID="ebe95fb3-e815-42fe-a591-aeee67dce92f" containerID="0caa60c2366915c30e68a1ab58b8e0e64dab9f7e00fe286ea553b558438ee0b5" exitCode=0 Oct 31 19:14:38.198704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:38.198380 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" event={"ID":"ebe95fb3-e815-42fe-a591-aeee67dce92f","Type":"ContainerDied","Data":"0caa60c2366915c30e68a1ab58b8e0e64dab9f7e00fe286ea553b558438ee0b5"} Oct 31 19:14:39.333866 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.333844 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:39.389113 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.389083 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca\") pod \"ebe95fb3-e815-42fe-a591-aeee67dce92f\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " Oct 31 19:14:39.389262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.389175 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca\") pod \"ebe95fb3-e815-42fe-a591-aeee67dce92f\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " Oct 31 19:14:39.389341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.389321 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx\" (UniqueName: \"kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx\") pod \"ebe95fb3-e815-42fe-a591-aeee67dce92f\" (UID: \"ebe95fb3-e815-42fe-a591-aeee67dce92f\") " Oct 31 19:14:39.389507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.389482 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca" pod "ebe95fb3-e815-42fe-a591-aeee67dce92f" (UID: "ebe95fb3-e815-42fe-a591-aeee67dce92f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:14:39.389678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.389660 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:14:39.391333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.391309 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca" pod "ebe95fb3-e815-42fe-a591-aeee67dce92f" (UID: "ebe95fb3-e815-42fe-a591-aeee67dce92f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:14:39.391456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.391439 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx" pod "ebe95fb3-e815-42fe-a591-aeee67dce92f" (UID: "ebe95fb3-e815-42fe-a591-aeee67dce92f"). InnerVolumeSpecName "kube-api-access-zkjbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:14:39.491241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.491214 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjbx\" (UniqueName: \"kubernetes.io/projected/ebe95fb3-e815-42fe-a591-aeee67dce92f-kube-api-access-zkjbx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:14:39.491241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:39.491238 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe95fb3-e815-42fe-a591-aeee67dce92f-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:14:40.211010 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:40.210982 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" Oct 31 19:14:40.211171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:40.211020 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2" event={"ID":"ebe95fb3-e815-42fe-a591-aeee67dce92f","Type":"ContainerDied","Data":"03f46e3f6d73a2ba4293a00018bdaa0cd53acb218f4ccf00f3b1b3c625f8ddfe"} Oct 31 19:14:40.211171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:40.211066 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f46e3f6d73a2ba4293a00018bdaa0cd53acb218f4ccf00f3b1b3c625f8ddfe" Oct 31 19:14:54.904853 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:14:54.904814 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:14:57.104818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.104782 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch"] Oct 31 19:14:57.105445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.105426 5688 cpu_manager.go:414] "containerMap: removing container" podUID="ebe95fb3-e815-42fe-a591-aeee67dce92f" containerName="revoke-keys" Oct 31 19:14:57.105502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.105450 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe95fb3-e815-42fe-a591-aeee67dce92f" containerName="revoke-keys" Oct 31 19:14:57.105640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.105627 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe95fb3-e815-42fe-a591-aeee67dce92f" containerName="revoke-keys" Oct 31 19:14:57.109260 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.109239 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.167566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.167539 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch"] Oct 31 19:14:57.212556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.212526 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d9922f\"/\"kube-root-ca.crt\"" Oct 31 19:14:57.212556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.212534 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d9922f\"/\"openshift-service-ca.crt\"" Oct 31 19:14:57.213151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.213134 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d9922f\"/\"explicit-model-d9922f-kserve-self-signed-certs\"" Oct 31 19:14:57.214178 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.214163 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d9922f\"/\"default-dockercfg-gsz7d\"" Oct 31 19:14:57.266532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.266490 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.266735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.266547 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.266735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.266643 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.266735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.266682 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.266735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.266732 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54h5n\" (UniqueName: \"kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.266970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.266886 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.367740 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.367651 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54h5n\" (UniqueName: \"kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.367914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.367793 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.367914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.367873 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.367914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.367902 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.368086 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.367929 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.368086 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.367962 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.368363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.368339 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.368506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.368367 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.368506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.368394 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.370538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.370517 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.370666 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.370618 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.428271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.428245 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54h5n\" (UniqueName: \"kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n\") pod \"explicit-model-d9922f-kserve-6fd7c6d887-hnbch\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:57.721947 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:57.721857 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:14:58.092926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:58.092901 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch"] Oct 31 19:14:58.094963 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:14:58.094931 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18556fd9_d936_40a1_9835_25216bf7f495.slice/crio-169e10102f5b5a700fc8c00be2995281c44549d401b1ef67982402e42774ae5d WatchSource:0}: Error finding container 169e10102f5b5a700fc8c00be2995281c44549d401b1ef67982402e42774ae5d: Status 404 returned error can't find the container with id 169e10102f5b5a700fc8c00be2995281c44549d401b1ef67982402e42774ae5d Oct 31 19:14:58.313000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:58.312961 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" event={"ID":"18556fd9-d936-40a1-9835-25216bf7f495","Type":"ContainerStarted","Data":"224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07"} Oct 31 19:14:58.313000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:58.313009 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" event={"ID":"18556fd9-d936-40a1-9835-25216bf7f495","Type":"ContainerStarted","Data":"169e10102f5b5a700fc8c00be2995281c44549d401b1ef67982402e42774ae5d"} Oct 31 19:14:58.429249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:58.429105 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" podStartSLOduration=1.429087402 podStartE2EDuration="1.429087402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:14:58.428005274 +0000 UTC m=+3219.030146417" watchObservedRunningTime="2026-10-31 19:14:58.429087402 +0000 UTC m=+3219.031228547" Oct 31 19:14:58.722151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:14:58.722076 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:15:00.164575 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.164547 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l"] Oct 31 19:15:00.168311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.168294 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6"] Oct 31 19:15:00.168450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.168429 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.171819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.171790 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm"] Oct 31 19:15:00.171950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.171916 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.175581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.175566 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.213861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.213816 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv"] Oct 31 19:15:00.217640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.217621 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl"] Oct 31 19:15:00.217769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.217753 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.221067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.221051 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w"] Oct 31 19:15:00.221213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.221153 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.221285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.221268 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:15:00.224705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.224685 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.263461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.263436 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm"] Oct 31 19:15:00.265265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.265247 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l"] Oct 31 19:15:00.266916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.266901 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6"] Oct 31 19:15:00.305212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305151 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tsm\" (UniqueName: \"kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.305330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305232 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxkx\" (UniqueName: \"kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.305330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305268 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.305330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305306 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74ms\" (UniqueName: \"kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.305493 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305377 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891235-sp4kl\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.305546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305494 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.305651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305630 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8dl\" (UniqueName: \"kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.305695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305669 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.305772 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305720 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfl52\" (UniqueName: \"kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52\") pod \"maas-api-key-cleanup-29891235-sp4kl\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.305772 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305761 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.305878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305814 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg75w\" (UniqueName: \"kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.305878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.305853 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.315956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.313606 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv"] Oct 31 19:15:00.368421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.368394 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w"] Oct 31 19:15:00.371968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.371938 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl"] Oct 31 19:15:00.406898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.406865 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tsm\" (UniqueName: \"kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.407056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.406903 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxkx\" (UniqueName: \"kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.407056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.406931 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.407056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.406964 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74ms\" (UniqueName: \"kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.407056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407020 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891235-sp4kl\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.407056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407047 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.407374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407081 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8dl\" (UniqueName: \"kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.407374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407108 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.407374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407139 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfl52\" (UniqueName: \"kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52\") pod \"maas-api-key-cleanup-29891235-sp4kl\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.407374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407220 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.407374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407285 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg75w\" (UniqueName: \"kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.407374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407326 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.407903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407877 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.407975 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.407935 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.408232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.408213 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.408334 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.408273 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891235-sp4kl\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.408417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.408401 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.408474 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.408434 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.465022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.464963 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfl52\" (UniqueName: \"kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52\") pod \"maas-api-key-cleanup-29891235-sp4kl\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.465252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.465234 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74ms\" (UniqueName: \"kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.465528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.465505 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg75w\" (UniqueName: \"kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.465601 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.465531 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tsm\" (UniqueName: \"kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.465601 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.465533 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8dl\" (UniqueName: \"kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.466294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.466276 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxkx\" (UniqueName: \"kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.484520 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.484495 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:00.498579 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.498561 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:00.505543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.505521 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:00.533312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.533283 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:00.539585 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.539555 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:00.545998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.545791 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:00.870072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.869846 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l"] Oct 31 19:15:00.872759 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:00.872732 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a93696_a204_4d0a_8a06_130b7005a245.slice/crio-a5a26479d46322ad6da6ac67e30b998e407dd8b2c8955f8ecd551ed068400f53 WatchSource:0}: Error finding container a5a26479d46322ad6da6ac67e30b998e407dd8b2c8955f8ecd551ed068400f53: Status 404 returned error can't find the container with id a5a26479d46322ad6da6ac67e30b998e407dd8b2c8955f8ecd551ed068400f53 Oct 31 19:15:00.923506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.923485 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm"] Oct 31 19:15:00.925064 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:00.925034 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfedae0_24b0_4030_9df4_82c43ca0f978.slice/crio-c2d751e73ebd7f856aa6b524d6af357b60f519ae96aff72c226ed12b7ec52562 WatchSource:0}: Error finding container c2d751e73ebd7f856aa6b524d6af357b60f519ae96aff72c226ed12b7ec52562: Status 404 returned error can't find the container with id c2d751e73ebd7f856aa6b524d6af357b60f519ae96aff72c226ed12b7ec52562 Oct 31 19:15:00.925680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.925550 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6"] Oct 31 19:15:00.925875 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:00.925837 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bd8e3f_4d21_4db9_90db_283242c18ee4.slice/crio-a86e1572017301b9a044f611e69fcabca9d93449cc606cebd8b02a4923bf62e9 WatchSource:0}: Error finding container a86e1572017301b9a044f611e69fcabca9d93449cc606cebd8b02a4923bf62e9: Status 404 returned error can't find the container with id a86e1572017301b9a044f611e69fcabca9d93449cc606cebd8b02a4923bf62e9 Oct 31 19:15:00.973383 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.973359 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv"] Oct 31 19:15:00.974879 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:00.974856 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4269af6a_5192_40da_a6b1_fa77eac68fcc.slice/crio-85c91ba095eb8ea764c9e6c5ad13e509fb704108f840d58aa4e2a759efae91e0 WatchSource:0}: Error finding container 85c91ba095eb8ea764c9e6c5ad13e509fb704108f840d58aa4e2a759efae91e0: Status 404 returned error can't find the container with id 85c91ba095eb8ea764c9e6c5ad13e509fb704108f840d58aa4e2a759efae91e0 Oct 31 19:15:00.992984 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.992946 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w"] Oct 31 19:15:00.995272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:00.995250 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl"] Oct 31 19:15:01.001404 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:01.001380 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b433f3_2474_4509_9c79_6dda2a1eb116.slice/crio-fc41fac741a7cb29ddb6e41aca16414588fd4195e94312da4ea14e3f05481d59 WatchSource:0}: Error finding container fc41fac741a7cb29ddb6e41aca16414588fd4195e94312da4ea14e3f05481d59: Status 404 returned error can't find the container with id fc41fac741a7cb29ddb6e41aca16414588fd4195e94312da4ea14e3f05481d59 Oct 31 19:15:01.001961 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:01.001944 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b6cf26_d0b2_431d_b2b3_20c7f7ceedd7.slice/crio-14136ced61218a2c9ecd7e6fc046509a2e123f48e88532f06a59d29213456d2b WatchSource:0}: Error finding container 14136ced61218a2c9ecd7e6fc046509a2e123f48e88532f06a59d29213456d2b: Status 404 returned error can't find the container with id 14136ced61218a2c9ecd7e6fc046509a2e123f48e88532f06a59d29213456d2b Oct 31 19:15:01.334032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.332586 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" event={"ID":"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7","Type":"ContainerStarted","Data":"14136ced61218a2c9ecd7e6fc046509a2e123f48e88532f06a59d29213456d2b"} Oct 31 19:15:01.335164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.335124 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerStarted","Data":"5de2982995010da0be815c8e0819578bfb1782a72354e105168423a12f2f0a49"} Oct 31 19:15:01.335334 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.335220 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerStarted","Data":"a5a26479d46322ad6da6ac67e30b998e407dd8b2c8955f8ecd551ed068400f53"} Oct 31 19:15:01.337140 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.337110 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" event={"ID":"c9bd8e3f-4d21-4db9-90db-283242c18ee4","Type":"ContainerStarted","Data":"a86e1572017301b9a044f611e69fcabca9d93449cc606cebd8b02a4923bf62e9"} Oct 31 19:15:01.338687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.338660 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" event={"ID":"37b433f3-2474-4509-9c79-6dda2a1eb116","Type":"ContainerStarted","Data":"fc41fac741a7cb29ddb6e41aca16414588fd4195e94312da4ea14e3f05481d59"} Oct 31 19:15:01.339971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.339947 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" event={"ID":"4269af6a-5192-40da-a6b1-fa77eac68fcc","Type":"ContainerStarted","Data":"85c91ba095eb8ea764c9e6c5ad13e509fb704108f840d58aa4e2a759efae91e0"} Oct 31 19:15:01.341687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.341664 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" event={"ID":"1dfedae0-24b0-4030-9df4-82c43ca0f978","Type":"ContainerStarted","Data":"e65eb2f9ca0b3b659ca5bbd0ef778777b277b9ec10991b9e9098f8e25d67ab69"} Oct 31 19:15:01.341754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.341717 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" event={"ID":"1dfedae0-24b0-4030-9df4-82c43ca0f978","Type":"ContainerStarted","Data":"c2d751e73ebd7f856aa6b524d6af357b60f519ae96aff72c226ed12b7ec52562"} Oct 31 19:15:01.450286 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.450241 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" podStartSLOduration=1.4502274339999999 podStartE2EDuration="1.450227434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:01.44726968 +0000 UTC m=+3222.049410818" watchObservedRunningTime="2026-10-31 19:15:01.450227434 +0000 UTC m=+3222.052368577" Oct 31 19:15:01.561997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:01.561913 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" podStartSLOduration=1.5618946889999998 podStartE2EDuration="1.561894689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:01.56007956 +0000 UTC m=+3222.162220706" watchObservedRunningTime="2026-10-31 19:15:01.561894689 +0000 UTC m=+3222.164035834" Oct 31 19:15:02.349237 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.349202 5688 generic.go:356] "Generic (PLEG): container finished" podID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerID="dd855eb3fff48226e5def244b0613c3dacd2c865a5ee6580678d423375092a87" exitCode=60 Oct 31 19:15:02.349658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.349272 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" event={"ID":"4269af6a-5192-40da-a6b1-fa77eac68fcc","Type":"ContainerDied","Data":"dd855eb3fff48226e5def244b0613c3dacd2c865a5ee6580678d423375092a87"} Oct 31 19:15:02.349658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.349528 5688 scope.go:122] "RemoveContainer" containerID="dd855eb3fff48226e5def244b0613c3dacd2c865a5ee6580678d423375092a87" Oct 31 19:15:02.351000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.350974 5688 generic.go:356] "Generic (PLEG): container finished" podID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerID="e65eb2f9ca0b3b659ca5bbd0ef778777b277b9ec10991b9e9098f8e25d67ab69" exitCode=60 Oct 31 19:15:02.351121 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.351055 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" event={"ID":"1dfedae0-24b0-4030-9df4-82c43ca0f978","Type":"ContainerDied","Data":"e65eb2f9ca0b3b659ca5bbd0ef778777b277b9ec10991b9e9098f8e25d67ab69"} Oct 31 19:15:02.351283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.351265 5688 scope.go:122] "RemoveContainer" containerID="e65eb2f9ca0b3b659ca5bbd0ef778777b277b9ec10991b9e9098f8e25d67ab69" Oct 31 19:15:02.352883 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.352860 5688 generic.go:356] "Generic (PLEG): container finished" podID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerID="033c174938e4ef1e271188fd58d52455bc14729182c50998a2b807c134dbdd7b" exitCode=60 Oct 31 19:15:02.352975 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.352936 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" event={"ID":"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7","Type":"ContainerDied","Data":"033c174938e4ef1e271188fd58d52455bc14729182c50998a2b807c134dbdd7b"} Oct 31 19:15:02.353151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.353135 5688 scope.go:122] "RemoveContainer" containerID="033c174938e4ef1e271188fd58d52455bc14729182c50998a2b807c134dbdd7b" Oct 31 19:15:02.354522 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.354502 5688 generic.go:356] "Generic (PLEG): container finished" podID="18a93696-a204-4d0a-8a06-130b7005a245" containerID="5de2982995010da0be815c8e0819578bfb1782a72354e105168423a12f2f0a49" exitCode=60 Oct 31 19:15:02.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.354573 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerDied","Data":"5de2982995010da0be815c8e0819578bfb1782a72354e105168423a12f2f0a49"} Oct 31 19:15:02.354755 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.354740 5688 scope.go:122] "RemoveContainer" containerID="5de2982995010da0be815c8e0819578bfb1782a72354e105168423a12f2f0a49" Oct 31 19:15:02.356019 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.356001 5688 generic.go:356] "Generic (PLEG): container finished" podID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerID="e2f3c56150ac57c0fb6f2aa49fb8c145337daeb9421f07c51b63047454a91fad" exitCode=60 Oct 31 19:15:02.356127 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.356067 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" event={"ID":"c9bd8e3f-4d21-4db9-90db-283242c18ee4","Type":"ContainerDied","Data":"e2f3c56150ac57c0fb6f2aa49fb8c145337daeb9421f07c51b63047454a91fad"} Oct 31 19:15:02.356257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.356243 5688 scope.go:122] "RemoveContainer" containerID="e2f3c56150ac57c0fb6f2aa49fb8c145337daeb9421f07c51b63047454a91fad" Oct 31 19:15:02.360737 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.360714 5688 generic.go:356] "Generic (PLEG): container finished" podID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerID="a4167fbd2153ea042c96b8350b18fa51c4215ad46c31e5463a810a4ca3b33bd3" exitCode=60 Oct 31 19:15:02.360839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.360772 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" event={"ID":"37b433f3-2474-4509-9c79-6dda2a1eb116","Type":"ContainerDied","Data":"a4167fbd2153ea042c96b8350b18fa51c4215ad46c31e5463a810a4ca3b33bd3"} Oct 31 19:15:02.360996 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:02.360981 5688 scope.go:122] "RemoveContainer" containerID="a4167fbd2153ea042c96b8350b18fa51c4215ad46c31e5463a810a4ca3b33bd3" Oct 31 19:15:03.370164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.370130 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" event={"ID":"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7","Type":"ContainerStarted","Data":"407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3"} Oct 31 19:15:03.372166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.372142 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerStarted","Data":"b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21"} Oct 31 19:15:03.374089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.374064 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" event={"ID":"c9bd8e3f-4d21-4db9-90db-283242c18ee4","Type":"ContainerStarted","Data":"feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d"} Oct 31 19:15:03.375897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.375875 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" event={"ID":"37b433f3-2474-4509-9c79-6dda2a1eb116","Type":"ContainerStarted","Data":"faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92"} Oct 31 19:15:03.377822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.377793 5688 generic.go:356] "Generic (PLEG): container finished" podID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerID="04dd40eaa4022cd9c3936d601464dc33b89fda2ad9d2f85b56341a2ed8f20ad8" exitCode=60 Oct 31 19:15:03.377927 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.377897 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" event={"ID":"4269af6a-5192-40da-a6b1-fa77eac68fcc","Type":"ContainerDied","Data":"04dd40eaa4022cd9c3936d601464dc33b89fda2ad9d2f85b56341a2ed8f20ad8"} Oct 31 19:15:03.377993 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.377938 5688 scope.go:122] "RemoveContainer" containerID="dd855eb3fff48226e5def244b0613c3dacd2c865a5ee6580678d423375092a87" Oct 31 19:15:03.378074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.378060 5688 scope.go:122] "RemoveContainer" containerID="04dd40eaa4022cd9c3936d601464dc33b89fda2ad9d2f85b56341a2ed8f20ad8" Oct 31 19:15:03.378305 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:03.378284 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv_odh-ai-gateway-infra(4269af6a-5192-40da-a6b1-fa77eac68fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" Oct 31 19:15:03.379959 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.379939 5688 generic.go:356] "Generic (PLEG): container finished" podID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerID="3e8e7da9da314abe2e68c96ea5837465292e3cc6cf72c542724db1549c0a2eba" exitCode=60 Oct 31 19:15:03.380056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.379970 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" event={"ID":"1dfedae0-24b0-4030-9df4-82c43ca0f978","Type":"ContainerDied","Data":"3e8e7da9da314abe2e68c96ea5837465292e3cc6cf72c542724db1549c0a2eba"} Oct 31 19:15:03.380291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.380269 5688 scope.go:122] "RemoveContainer" containerID="3e8e7da9da314abe2e68c96ea5837465292e3cc6cf72c542724db1549c0a2eba" Oct 31 19:15:03.380533 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:03.380512 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6_odh-ai-gateway-infra(1dfedae0-24b0-4030-9df4-82c43ca0f978)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" Oct 31 19:15:03.447532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.446494 5688 scope.go:122] "RemoveContainer" containerID="e65eb2f9ca0b3b659ca5bbd0ef778777b277b9ec10991b9e9098f8e25d67ab69" Oct 31 19:15:03.483639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.483597 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" podStartSLOduration=3.483584356 podStartE2EDuration="3.483584356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:03.483007271 +0000 UTC m=+3224.085148415" watchObservedRunningTime="2026-10-31 19:15:03.483584356 +0000 UTC m=+3224.085725497" Oct 31 19:15:03.598262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.598145 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" podStartSLOduration=3.59812751 podStartE2EDuration="3.59812751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:03.597454616 +0000 UTC m=+3224.199595762" watchObservedRunningTime="2026-10-31 19:15:03.59812751 +0000 UTC m=+3224.200268654" Oct 31 19:15:03.608280 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:03.608243 5688 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a93696_a204_4d0a_8a06_130b7005a245.slice/crio-conmon-b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bd8e3f_4d21_4db9_90db_283242c18ee4.slice/crio-conmon-feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a93696_a204_4d0a_8a06_130b7005a245.slice/crio-b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bd8e3f_4d21_4db9_90db_283242c18ee4.slice/crio-feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d.scope\": RecentStats: unable to find data in memory cache]" Oct 31 19:15:03.823703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:03.823654 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" podStartSLOduration=3.823640169 podStartE2EDuration="3.823640169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:03.822558235 +0000 UTC m=+3224.424699372" watchObservedRunningTime="2026-10-31 19:15:03.823640169 +0000 UTC m=+3224.425781313" Oct 31 19:15:04.417948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.416658 5688 scope.go:122] "RemoveContainer" containerID="04dd40eaa4022cd9c3936d601464dc33b89fda2ad9d2f85b56341a2ed8f20ad8" Oct 31 19:15:04.422065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.422030 5688 scope.go:122] "RemoveContainer" containerID="3e8e7da9da314abe2e68c96ea5837465292e3cc6cf72c542724db1549c0a2eba" Oct 31 19:15:04.423327 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:04.423296 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv_odh-ai-gateway-infra(4269af6a-5192-40da-a6b1-fa77eac68fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" Oct 31 19:15:04.423991 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:04.423967 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6_odh-ai-gateway-infra(1dfedae0-24b0-4030-9df4-82c43ca0f978)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" Oct 31 19:15:04.432588 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.432561 5688 generic.go:356] "Generic (PLEG): container finished" podID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerID="407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3" exitCode=60 Oct 31 19:15:04.432756 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.432726 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" event={"ID":"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7","Type":"ContainerDied","Data":"407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3"} Oct 31 19:15:04.432856 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.432836 5688 scope.go:122] "RemoveContainer" containerID="033c174938e4ef1e271188fd58d52455bc14729182c50998a2b807c134dbdd7b" Oct 31 19:15:04.432912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.432897 5688 scope.go:122] "RemoveContainer" containerID="407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3" Oct 31 19:15:04.433196 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:04.433156 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w_odh-ai-gateway-infra(c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" Oct 31 19:15:04.435388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.435366 5688 generic.go:356] "Generic (PLEG): container finished" podID="18a93696-a204-4d0a-8a06-130b7005a245" containerID="b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21" exitCode=60 Oct 31 19:15:04.435519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.435456 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerDied","Data":"b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21"} Oct 31 19:15:04.435693 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.435662 5688 scope.go:122] "RemoveContainer" containerID="b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21" Oct 31 19:15:04.435963 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:04.435940 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l_odh-ai-gateway-infra(18a93696-a204-4d0a-8a06-130b7005a245)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" podUID="18a93696-a204-4d0a-8a06-130b7005a245" Oct 31 19:15:04.438092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.438072 5688 generic.go:356] "Generic (PLEG): container finished" podID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerID="feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d" exitCode=60 Oct 31 19:15:04.438240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.438166 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" event={"ID":"c9bd8e3f-4d21-4db9-90db-283242c18ee4","Type":"ContainerDied","Data":"feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d"} Oct 31 19:15:04.438445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.438416 5688 scope.go:122] "RemoveContainer" containerID="feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d" Oct 31 19:15:04.438694 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:04.438663 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm_odh-ai-gateway-infra(c9bd8e3f-4d21-4db9-90db-283242c18ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" Oct 31 19:15:04.440243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.440224 5688 generic.go:356] "Generic (PLEG): container finished" podID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerID="faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92" exitCode=60 Oct 31 19:15:04.440338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.440297 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" event={"ID":"37b433f3-2474-4509-9c79-6dda2a1eb116","Type":"ContainerDied","Data":"faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92"} Oct 31 19:15:04.440611 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.440593 5688 scope.go:122] "RemoveContainer" containerID="faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92" Oct 31 19:15:04.440799 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:04.440783 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891235-sp4kl_odh-ai-gateway-infra(37b433f3-2474-4509-9c79-6dda2a1eb116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" Oct 31 19:15:04.446857 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.446839 5688 scope.go:122] "RemoveContainer" containerID="5de2982995010da0be815c8e0819578bfb1782a72354e105168423a12f2f0a49" Oct 31 19:15:04.460017 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.459995 5688 scope.go:122] "RemoveContainer" containerID="e2f3c56150ac57c0fb6f2aa49fb8c145337daeb9421f07c51b63047454a91fad" Oct 31 19:15:04.472938 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:04.472916 5688 scope.go:122] "RemoveContainer" containerID="a4167fbd2153ea042c96b8350b18fa51c4215ad46c31e5463a810a4ca3b33bd3" Oct 31 19:15:05.449692 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:05.449664 5688 scope.go:122] "RemoveContainer" containerID="407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3" Oct 31 19:15:05.450157 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:05.449973 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w_odh-ai-gateway-infra(c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" Oct 31 19:15:05.451456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:05.451429 5688 scope.go:122] "RemoveContainer" containerID="b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21" Oct 31 19:15:05.451731 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:05.451712 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l_odh-ai-gateway-infra(18a93696-a204-4d0a-8a06-130b7005a245)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" podUID="18a93696-a204-4d0a-8a06-130b7005a245" Oct 31 19:15:05.453099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:05.453078 5688 scope.go:122] "RemoveContainer" containerID="feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d" Oct 31 19:15:05.453326 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:05.453308 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm_odh-ai-gateway-infra(c9bd8e3f-4d21-4db9-90db-283242c18ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" Oct 31 19:15:05.454929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:05.454910 5688 scope.go:122] "RemoveContainer" containerID="faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92" Oct 31 19:15:05.455135 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:05.455116 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891235-sp4kl_odh-ai-gateway-infra(37b433f3-2474-4509-9c79-6dda2a1eb116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" Oct 31 19:15:07.722986 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:07.722956 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:15:07.736164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:07.736138 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:15:08.482942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:08.482917 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:15:10.649473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.649443 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n"] Oct 31 19:15:10.654419 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.654404 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.706700 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.706678 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:15:10.724052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.724025 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.724257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.724146 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.724257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.724224 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.758506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.758481 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n"] Oct 31 19:15:10.825061 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.825035 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.825166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.825097 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.825166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.825128 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.825726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.825710 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.827546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.827526 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.881244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.881224 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw\") pod \"maas-api-revoke-sub-38a50397-8w67n\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:10.966794 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:10.966736 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:11.339886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:11.339861 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n"] Oct 31 19:15:11.342235 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:11.342205 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d735f3_d157_4ef6_8aeb_5f9781f2b2c8.slice/crio-5a70426d1faf21e6a25fcedec801ad78bef61d6bce46da96d86d6733913e3b63 WatchSource:0}: Error finding container 5a70426d1faf21e6a25fcedec801ad78bef61d6bce46da96d86d6733913e3b63: Status 404 returned error can't find the container with id 5a70426d1faf21e6a25fcedec801ad78bef61d6bce46da96d86d6733913e3b63 Oct 31 19:15:11.490677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:11.490643 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" event={"ID":"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8","Type":"ContainerStarted","Data":"5a70426d1faf21e6a25fcedec801ad78bef61d6bce46da96d86d6733913e3b63"} Oct 31 19:15:12.505401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:12.505368 5688 generic.go:356] "Generic (PLEG): container finished" podID="22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" containerID="8791c31b22ce4ba8c15c27fa6381287431ddc61163d083588b0f6e5ead2d12c0" exitCode=0 Oct 31 19:15:12.505785 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:12.505412 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" event={"ID":"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8","Type":"ContainerDied","Data":"8791c31b22ce4ba8c15c27fa6381287431ddc61163d083588b0f6e5ead2d12c0"} Oct 31 19:15:13.655798 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.655775 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:13.756988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.756958 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw\") pod \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " Oct 31 19:15:13.757122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.757083 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca\") pod \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " Oct 31 19:15:13.757207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.757136 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca\") pod \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\" (UID: \"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8\") " Oct 31 19:15:13.757483 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.757459 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca" pod "22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" (UID: "22d735f3-d157-4ef6-8aeb-5f9781f2b2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:13.759106 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.759079 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw" pod "22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" (UID: "22d735f3-d157-4ef6-8aeb-5f9781f2b2c8"). InnerVolumeSpecName "kube-api-access-jwxgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:13.759241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.759218 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca" pod "22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" (UID: "22d735f3-d157-4ef6-8aeb-5f9781f2b2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:15:13.858644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.858617 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:13.858644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.858644 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:13.858812 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:13.858653 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8-kube-api-access-jwxgw\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:14.517449 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:14.517412 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" event={"ID":"22d735f3-d157-4ef6-8aeb-5f9781f2b2c8","Type":"ContainerDied","Data":"5a70426d1faf21e6a25fcedec801ad78bef61d6bce46da96d86d6733913e3b63"} Oct 31 19:15:14.517639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:14.517474 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a70426d1faf21e6a25fcedec801ad78bef61d6bce46da96d86d6733913e3b63" Oct 31 19:15:14.517639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:14.517427 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n" Oct 31 19:15:14.558830 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:14.558807 5688 scope.go:122] "RemoveContainer" containerID="407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3" Oct 31 19:15:14.558942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:14.558931 5688 scope.go:122] "RemoveContainer" containerID="feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d" Oct 31 19:15:15.528800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.528776 5688 generic.go:356] "Generic (PLEG): container finished" podID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerID="f24a329d06e8262ae1aef8958a58bb709e442bca83eccb0396d87c743d52f5c6" exitCode=60 Oct 31 19:15:15.529255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.528854 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" event={"ID":"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7","Type":"ContainerDied","Data":"f24a329d06e8262ae1aef8958a58bb709e442bca83eccb0396d87c743d52f5c6"} Oct 31 19:15:15.529255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.528933 5688 scope.go:122] "RemoveContainer" containerID="407e7a99dea64d51124a04c32bfff6d679fc466ef6b5cb1a3e4d729293bdbbb3" Oct 31 19:15:15.529373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.529258 5688 scope.go:122] "RemoveContainer" containerID="f24a329d06e8262ae1aef8958a58bb709e442bca83eccb0396d87c743d52f5c6" Oct 31 19:15:15.529551 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:15.529526 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w_odh-ai-gateway-infra(c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" Oct 31 19:15:15.530777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.530758 5688 generic.go:356] "Generic (PLEG): container finished" podID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerID="ac2f0d17ff0001535b0346cf166f3f5e3294be7530c00116360a5b34cfba845f" exitCode=60 Oct 31 19:15:15.530886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.530835 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" event={"ID":"c9bd8e3f-4d21-4db9-90db-283242c18ee4","Type":"ContainerDied","Data":"ac2f0d17ff0001535b0346cf166f3f5e3294be7530c00116360a5b34cfba845f"} Oct 31 19:15:15.531162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.531146 5688 scope.go:122] "RemoveContainer" containerID="ac2f0d17ff0001535b0346cf166f3f5e3294be7530c00116360a5b34cfba845f" Oct 31 19:15:15.531428 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:15.531409 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm_odh-ai-gateway-infra(c9bd8e3f-4d21-4db9-90db-283242c18ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" Oct 31 19:15:15.559633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.559615 5688 scope.go:122] "RemoveContainer" containerID="faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92" Oct 31 19:15:15.562356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:15.562334 5688 scope.go:122] "RemoveContainer" containerID="feef39b44b0d57e65a480955250b84d9468087e52a55ffe2734a623a9e4fba7d" Oct 31 19:15:16.541840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.541814 5688 generic.go:356] "Generic (PLEG): container finished" podID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerID="431a5c4a9f4d4cb34ffc0a5fbf1b971a92cdfc65757da7d706ecad76622cbfa6" exitCode=60 Oct 31 19:15:16.542244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.541885 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" event={"ID":"37b433f3-2474-4509-9c79-6dda2a1eb116","Type":"ContainerDied","Data":"431a5c4a9f4d4cb34ffc0a5fbf1b971a92cdfc65757da7d706ecad76622cbfa6"} Oct 31 19:15:16.542244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.541955 5688 scope.go:122] "RemoveContainer" containerID="faf6df9dcb49cf65f9a45d6982cc5ea7c4ab4db00f44d6ca1c954b915f8b0a92" Oct 31 19:15:16.542361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.542346 5688 scope.go:122] "RemoveContainer" containerID="431a5c4a9f4d4cb34ffc0a5fbf1b971a92cdfc65757da7d706ecad76622cbfa6" Oct 31 19:15:16.542594 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:16.542577 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891235-sp4kl_odh-ai-gateway-infra(37b433f3-2474-4509-9c79-6dda2a1eb116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" Oct 31 19:15:16.559205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.559170 5688 scope.go:122] "RemoveContainer" containerID="b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21" Oct 31 19:15:16.653745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.653714 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w"] Oct 31 19:15:16.764555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.764527 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm"] Oct 31 19:15:16.820451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.820425 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:16.893909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.893883 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm\" (UniqueName: \"kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm\") pod \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " Oct 31 19:15:16.894005 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.893977 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca\") pod \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\" (UID: \"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7\") " Oct 31 19:15:16.894352 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.894330 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca" pod "c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" (UID: "c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:16.894551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.894536 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:16.896037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.896013 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm" pod "c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" (UID: "c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7"). InnerVolumeSpecName "kube-api-access-25tsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:16.912718 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.912699 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:16.995148 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.995119 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms\" (UniqueName: \"kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms\") pod \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " Oct 31 19:15:16.995290 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.995265 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca\") pod \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\" (UID: \"c9bd8e3f-4d21-4db9-90db-283242c18ee4\") " Oct 31 19:15:16.995871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.995632 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca" pod "c9bd8e3f-4d21-4db9-90db-283242c18ee4" (UID: "c9bd8e3f-4d21-4db9-90db-283242c18ee4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:16.995871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.995709 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tsm\" (UniqueName: \"kubernetes.io/projected/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7-kube-api-access-25tsm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:16.997687 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:16.997667 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms" pod "c9bd8e3f-4d21-4db9-90db-283242c18ee4" (UID: "c9bd8e3f-4d21-4db9-90db-283242c18ee4"). InnerVolumeSpecName "kube-api-access-d74ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:17.097041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.097011 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bd8e3f-4d21-4db9-90db-283242c18ee4-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:17.097041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.097035 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74ms\" (UniqueName: \"kubernetes.io/projected/c9bd8e3f-4d21-4db9-90db-283242c18ee4-kube-api-access-d74ms\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:17.548855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.548829 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" Oct 31 19:15:17.549291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.548826 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w" event={"ID":"c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7","Type":"ContainerDied","Data":"14136ced61218a2c9ecd7e6fc046509a2e123f48e88532f06a59d29213456d2b"} Oct 31 19:15:17.549291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.548976 5688 scope.go:122] "RemoveContainer" containerID="f24a329d06e8262ae1aef8958a58bb709e442bca83eccb0396d87c743d52f5c6" Oct 31 19:15:17.550943 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.550915 5688 generic.go:356] "Generic (PLEG): container finished" podID="18a93696-a204-4d0a-8a06-130b7005a245" containerID="f5a39e75cd9e0ed6f8ad9b842784b0fb34d472589401315a02d2f4d75385ee23" exitCode=60 Oct 31 19:15:17.551056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.550937 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerDied","Data":"f5a39e75cd9e0ed6f8ad9b842784b0fb34d472589401315a02d2f4d75385ee23"} Oct 31 19:15:17.551339 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.551322 5688 scope.go:122] "RemoveContainer" containerID="f5a39e75cd9e0ed6f8ad9b842784b0fb34d472589401315a02d2f4d75385ee23" Oct 31 19:15:17.551608 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:17.551576 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l_odh-ai-gateway-infra(18a93696-a204-4d0a-8a06-130b7005a245)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" podUID="18a93696-a204-4d0a-8a06-130b7005a245" Oct 31 19:15:17.552553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.552536 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" Oct 31 19:15:17.552664 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.552566 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm" event={"ID":"c9bd8e3f-4d21-4db9-90db-283242c18ee4","Type":"ContainerDied","Data":"a86e1572017301b9a044f611e69fcabca9d93449cc606cebd8b02a4923bf62e9"} Oct 31 19:15:17.563327 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.563310 5688 scope.go:122] "RemoveContainer" containerID="b04f0c06b71ba702608b7f2a7808972c9356b90d5186a5baba4ce9a39804ca21" Oct 31 19:15:17.564004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.563983 5688 scope.go:122] "RemoveContainer" containerID="04dd40eaa4022cd9c3936d601464dc33b89fda2ad9d2f85b56341a2ed8f20ad8" Oct 31 19:15:17.564115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.564048 5688 scope.go:122] "RemoveContainer" containerID="3e8e7da9da314abe2e68c96ea5837465292e3cc6cf72c542724db1549c0a2eba" Oct 31 19:15:17.582979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.582960 5688 scope.go:122] "RemoveContainer" containerID="ac2f0d17ff0001535b0346cf166f3f5e3294be7530c00116360a5b34cfba845f" Oct 31 19:15:17.666313 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.666289 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl"] Oct 31 19:15:17.716861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.716829 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w"] Oct 31 19:15:17.722535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.722509 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891235-xxz6w"] Oct 31 19:15:17.814795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.814770 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:17.905984 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.905918 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca\") pod \"37b433f3-2474-4509-9c79-6dda2a1eb116\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " Oct 31 19:15:17.906099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.906039 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52\" (UniqueName: \"kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52\") pod \"37b433f3-2474-4509-9c79-6dda2a1eb116\" (UID: \"37b433f3-2474-4509-9c79-6dda2a1eb116\") " Oct 31 19:15:17.906245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.906225 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca" pod "37b433f3-2474-4509-9c79-6dda2a1eb116" (UID: "37b433f3-2474-4509-9c79-6dda2a1eb116"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:17.906354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.906340 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b433f3-2474-4509-9c79-6dda2a1eb116-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:17.907984 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.907966 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52" pod "37b433f3-2474-4509-9c79-6dda2a1eb116" (UID: "37b433f3-2474-4509-9c79-6dda2a1eb116"). InnerVolumeSpecName "kube-api-access-wfl52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:17.941401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.941376 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm"] Oct 31 19:15:17.946286 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:17.946262 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891235-fxfcm"] Oct 31 19:15:18.007455 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.007433 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfl52\" (UniqueName: \"kubernetes.io/projected/37b433f3-2474-4509-9c79-6dda2a1eb116-kube-api-access-wfl52\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:18.563390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.563363 5688 generic.go:356] "Generic (PLEG): container finished" podID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerID="d991d3de341bdad4a2eac315959bd78da9452a1899a54d6dc7f4782fa3ccf91b" exitCode=60 Oct 31 19:15:18.563876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.563456 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" event={"ID":"4269af6a-5192-40da-a6b1-fa77eac68fcc","Type":"ContainerDied","Data":"d991d3de341bdad4a2eac315959bd78da9452a1899a54d6dc7f4782fa3ccf91b"} Oct 31 19:15:18.563876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.563573 5688 scope.go:122] "RemoveContainer" containerID="04dd40eaa4022cd9c3936d601464dc33b89fda2ad9d2f85b56341a2ed8f20ad8" Oct 31 19:15:18.564016 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.563877 5688 scope.go:122] "RemoveContainer" containerID="d991d3de341bdad4a2eac315959bd78da9452a1899a54d6dc7f4782fa3ccf91b" Oct 31 19:15:18.564156 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:18.564124 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv_odh-ai-gateway-infra(4269af6a-5192-40da-a6b1-fa77eac68fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" Oct 31 19:15:18.565945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.565925 5688 generic.go:356] "Generic (PLEG): container finished" podID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerID="4eb7846311c6a28d3e0695665143bca4884a214715893d53ea72007a7380ee7f" exitCode=60 Oct 31 19:15:18.566031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.566002 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" event={"ID":"1dfedae0-24b0-4030-9df4-82c43ca0f978","Type":"ContainerDied","Data":"4eb7846311c6a28d3e0695665143bca4884a214715893d53ea72007a7380ee7f"} Oct 31 19:15:18.566482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.566460 5688 scope.go:122] "RemoveContainer" containerID="4eb7846311c6a28d3e0695665143bca4884a214715893d53ea72007a7380ee7f" Oct 31 19:15:18.566752 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:18.566729 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6_odh-ai-gateway-infra(1dfedae0-24b0-4030-9df4-82c43ca0f978)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" Oct 31 19:15:18.571227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.571176 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" Oct 31 19:15:18.571301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.571237 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl" event={"ID":"37b433f3-2474-4509-9c79-6dda2a1eb116","Type":"ContainerDied","Data":"fc41fac741a7cb29ddb6e41aca16414588fd4195e94312da4ea14e3f05481d59"} Oct 31 19:15:18.579323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.579304 5688 scope.go:122] "RemoveContainer" containerID="3e8e7da9da314abe2e68c96ea5837465292e3cc6cf72c542724db1549c0a2eba" Oct 31 19:15:18.591132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.591117 5688 scope.go:122] "RemoveContainer" containerID="431a5c4a9f4d4cb34ffc0a5fbf1b971a92cdfc65757da7d706ecad76622cbfa6" Oct 31 19:15:18.786514 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.786481 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl"] Oct 31 19:15:18.788764 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.788738 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891235-sp4kl"] Oct 31 19:15:18.841846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:18.841816 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l"] Oct 31 19:15:19.001388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.001368 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:19.118409 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.118383 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca\") pod \"18a93696-a204-4d0a-8a06-130b7005a245\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " Oct 31 19:15:19.118561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.118444 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl\" (UniqueName: \"kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl\") pod \"18a93696-a204-4d0a-8a06-130b7005a245\" (UID: \"18a93696-a204-4d0a-8a06-130b7005a245\") " Oct 31 19:15:19.118778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.118752 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca" pod "18a93696-a204-4d0a-8a06-130b7005a245" (UID: "18a93696-a204-4d0a-8a06-130b7005a245"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:19.118978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.118965 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a93696-a204-4d0a-8a06-130b7005a245-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:19.120671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.120648 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl" pod "18a93696-a204-4d0a-8a06-130b7005a245" (UID: "18a93696-a204-4d0a-8a06-130b7005a245"). InnerVolumeSpecName "kube-api-access-kz8dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:19.220249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.220155 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8dl\" (UniqueName: \"kubernetes.io/projected/18a93696-a204-4d0a-8a06-130b7005a245-kube-api-access-kz8dl\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:19.585813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.585779 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" path="/var/lib/kubelet/pods/37b433f3-2474-4509-9c79-6dda2a1eb116/volumes" Oct 31 19:15:19.586320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.586143 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" path="/var/lib/kubelet/pods/c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7/volumes" Oct 31 19:15:19.586544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.586528 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" path="/var/lib/kubelet/pods/c9bd8e3f-4d21-4db9-90db-283242c18ee4/volumes" Oct 31 19:15:19.595628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.595605 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" Oct 31 19:15:19.595771 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.595630 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l" event={"ID":"18a93696-a204-4d0a-8a06-130b7005a245","Type":"ContainerDied","Data":"a5a26479d46322ad6da6ac67e30b998e407dd8b2c8955f8ecd551ed068400f53"} Oct 31 19:15:19.595771 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.595690 5688 scope.go:122] "RemoveContainer" containerID="f5a39e75cd9e0ed6f8ad9b842784b0fb34d472589401315a02d2f4d75385ee23" Oct 31 19:15:19.684882 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.684853 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv"] Oct 31 19:15:19.768703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.768666 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l"] Oct 31 19:15:19.771033 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.770919 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891235-jzb7l"] Oct 31 19:15:19.832964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.832937 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:19.913173 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.913098 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6"] Oct 31 19:15:19.926927 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.926898 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx\" (UniqueName: \"kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx\") pod \"4269af6a-5192-40da-a6b1-fa77eac68fcc\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " Oct 31 19:15:19.927101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.927016 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca\") pod \"4269af6a-5192-40da-a6b1-fa77eac68fcc\" (UID: \"4269af6a-5192-40da-a6b1-fa77eac68fcc\") " Oct 31 19:15:19.928093 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.928062 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca" pod "4269af6a-5192-40da-a6b1-fa77eac68fcc" (UID: "4269af6a-5192-40da-a6b1-fa77eac68fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:19.929382 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:19.929353 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx" pod "4269af6a-5192-40da-a6b1-fa77eac68fcc" (UID: "4269af6a-5192-40da-a6b1-fa77eac68fcc"). InnerVolumeSpecName "kube-api-access-kwxkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:20.028667 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.028632 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4269af6a-5192-40da-a6b1-fa77eac68fcc-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:20.028814 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.028670 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxkx\" (UniqueName: \"kubernetes.io/projected/4269af6a-5192-40da-a6b1-fa77eac68fcc-kube-api-access-kwxkx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:20.063617 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.063594 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:20.129275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.129245 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca\") pod \"1dfedae0-24b0-4030-9df4-82c43ca0f978\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " Oct 31 19:15:20.129445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.129416 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w\" (UniqueName: \"kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w\") pod \"1dfedae0-24b0-4030-9df4-82c43ca0f978\" (UID: \"1dfedae0-24b0-4030-9df4-82c43ca0f978\") " Oct 31 19:15:20.129621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.129599 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca" pod "1dfedae0-24b0-4030-9df4-82c43ca0f978" (UID: "1dfedae0-24b0-4030-9df4-82c43ca0f978"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:20.129768 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.129752 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfedae0-24b0-4030-9df4-82c43ca0f978-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:20.131556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.131538 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w" pod "1dfedae0-24b0-4030-9df4-82c43ca0f978" (UID: "1dfedae0-24b0-4030-9df4-82c43ca0f978"). InnerVolumeSpecName "kube-api-access-mg75w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:20.230809 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.230749 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg75w\" (UniqueName: \"kubernetes.io/projected/1dfedae0-24b0-4030-9df4-82c43ca0f978-kube-api-access-mg75w\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:20.604018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.603993 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" Oct 31 19:15:20.604018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.604006 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv" event={"ID":"4269af6a-5192-40da-a6b1-fa77eac68fcc","Type":"ContainerDied","Data":"85c91ba095eb8ea764c9e6c5ad13e509fb704108f840d58aa4e2a759efae91e0"} Oct 31 19:15:20.604561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.604049 5688 scope.go:122] "RemoveContainer" containerID="d991d3de341bdad4a2eac315959bd78da9452a1899a54d6dc7f4782fa3ccf91b" Oct 31 19:15:20.605515 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.605494 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" Oct 31 19:15:20.605613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.605522 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6" event={"ID":"1dfedae0-24b0-4030-9df4-82c43ca0f978","Type":"ContainerDied","Data":"c2d751e73ebd7f856aa6b524d6af357b60f519ae96aff72c226ed12b7ec52562"} Oct 31 19:15:20.613709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.613691 5688 scope.go:122] "RemoveContainer" containerID="4eb7846311c6a28d3e0695665143bca4884a214715893d53ea72007a7380ee7f" Oct 31 19:15:20.766742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.766713 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv"] Oct 31 19:15:20.771974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.771950 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891235-tpklv"] Oct 31 19:15:20.879514 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.879453 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6"] Oct 31 19:15:20.883217 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:20.883178 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891235-z4zq6"] Oct 31 19:15:21.376478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376454 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs"] Oct 31 19:15:21.376891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376879 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:15:21.376931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376893 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:15:21.376931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376905 5688 cpu_manager.go:414] "containerMap: removing container" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.376931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376911 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.376931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376919 5688 cpu_manager.go:414] "containerMap: removing container" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.376931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376925 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376936 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376941 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376952 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376956 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376965 5688 cpu_manager.go:414] "containerMap: removing container" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376969 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376979 5688 cpu_manager.go:414] "containerMap: removing container" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376983 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376990 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.376995 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377002 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377006 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377013 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377017 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377022 5688 cpu_manager.go:414] "containerMap: removing container" podUID="22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" containerName="revoke-keys" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377026 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" containerName="revoke-keys" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377039 5688 cpu_manager.go:414] "containerMap: removing container" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377044 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377057 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377061 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377070 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377074 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377081 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.377104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377085 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377158 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377165 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377171 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377177 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377197 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377203 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377208 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377213 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377218 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377223 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377227 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377232 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" containerName="revoke-keys" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377238 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b433f3-2474-4509-9c79-6dda2a1eb116" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377243 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377248 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377253 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:15:21.377879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.377258 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:15:21.381741 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.381723 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.434449 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.434426 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:15:21.434557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.434461 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:15:21.437168 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.437143 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs"] Oct 31 19:15:21.442222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.442177 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.442320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.442273 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sdt\" (UniqueName: \"kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.442371 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.442338 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.542878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.542804 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.543014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.542907 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.543014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.542928 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sdt\" (UniqueName: \"kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.543536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.543503 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.545227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.545206 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.566467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.566443 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a93696-a204-4d0a-8a06-130b7005a245" path="/var/lib/kubelet/pods/18a93696-a204-4d0a-8a06-130b7005a245/volumes" Oct 31 19:15:21.566832 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.566819 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfedae0-24b0-4030-9df4-82c43ca0f978" path="/var/lib/kubelet/pods/1dfedae0-24b0-4030-9df4-82c43ca0f978/volumes" Oct 31 19:15:21.567172 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.567158 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" path="/var/lib/kubelet/pods/4269af6a-5192-40da-a6b1-fa77eac68fcc/volumes" Oct 31 19:15:21.600456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.600431 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sdt\" (UniqueName: \"kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt\") pod \"maas-api-revoke-sub-f5464e64-b8mcs\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:21.695104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:21.695083 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:22.067127 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:22.067105 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs"] Oct 31 19:15:22.072926 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:15:22.069816 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0da11b_3490_452c_8c1c_b346274e32c3.slice/crio-85429e2f3bd7392a7274a82cee1d29ec3b3bb4af15dd6613d21e7f59cdee1405 WatchSource:0}: Error finding container 85429e2f3bd7392a7274a82cee1d29ec3b3bb4af15dd6613d21e7f59cdee1405: Status 404 returned error can't find the container with id 85429e2f3bd7392a7274a82cee1d29ec3b3bb4af15dd6613d21e7f59cdee1405 Oct 31 19:15:22.628071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:22.627973 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" event={"ID":"8a0da11b-3490-452c-8c1c-b346274e32c3","Type":"ContainerStarted","Data":"7f3d288836ece63c7bfbe3bbbfd1d4ca88a0c6abface0e372cf00a4ff668cde1"} Oct 31 19:15:22.628071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:22.628056 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" event={"ID":"8a0da11b-3490-452c-8c1c-b346274e32c3","Type":"ContainerStarted","Data":"85429e2f3bd7392a7274a82cee1d29ec3b3bb4af15dd6613d21e7f59cdee1405"} Oct 31 19:15:22.741926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:22.741879 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" podStartSLOduration=1.741866204 podStartE2EDuration="1.741866204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:22.740013205 +0000 UTC m=+3243.342154349" watchObservedRunningTime="2026-10-31 19:15:22.741866204 +0000 UTC m=+3243.344007348" Oct 31 19:15:23.636061 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:23.636033 5688 generic.go:356] "Generic (PLEG): container finished" podID="8a0da11b-3490-452c-8c1c-b346274e32c3" containerID="7f3d288836ece63c7bfbe3bbbfd1d4ca88a0c6abface0e372cf00a4ff668cde1" exitCode=0 Oct 31 19:15:23.636246 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:23.636107 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" event={"ID":"8a0da11b-3490-452c-8c1c-b346274e32c3","Type":"ContainerDied","Data":"7f3d288836ece63c7bfbe3bbbfd1d4ca88a0c6abface0e372cf00a4ff668cde1"} Oct 31 19:15:24.789477 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.789455 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:24.875860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.875834 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca\") pod \"8a0da11b-3490-452c-8c1c-b346274e32c3\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " Oct 31 19:15:24.875968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.875868 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca\") pod \"8a0da11b-3490-452c-8c1c-b346274e32c3\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " Oct 31 19:15:24.875968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.875913 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt\" (UniqueName: \"kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt\") pod \"8a0da11b-3490-452c-8c1c-b346274e32c3\" (UID: \"8a0da11b-3490-452c-8c1c-b346274e32c3\") " Oct 31 19:15:24.876267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.876240 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca" pod "8a0da11b-3490-452c-8c1c-b346274e32c3" (UID: "8a0da11b-3490-452c-8c1c-b346274e32c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:15:24.878136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.878107 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt" pod "8a0da11b-3490-452c-8c1c-b346274e32c3" (UID: "8a0da11b-3490-452c-8c1c-b346274e32c3"). InnerVolumeSpecName "kube-api-access-g5sdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:24.878262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.878146 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca" pod "8a0da11b-3490-452c-8c1c-b346274e32c3" (UID: "8a0da11b-3490-452c-8c1c-b346274e32c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:15:24.905990 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:24.905919 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:15:24.976776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.976749 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:24.976776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.976773 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0da11b-3490-452c-8c1c-b346274e32c3-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:24.976932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:24.976783 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5sdt\" (UniqueName: \"kubernetes.io/projected/8a0da11b-3490-452c-8c1c-b346274e32c3-kube-api-access-g5sdt\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:25.647999 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:25.647967 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" event={"ID":"8a0da11b-3490-452c-8c1c-b346274e32c3","Type":"ContainerDied","Data":"85429e2f3bd7392a7274a82cee1d29ec3b3bb4af15dd6613d21e7f59cdee1405"} Oct 31 19:15:25.647999 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:25.647992 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs" Oct 31 19:15:25.648174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:25.648022 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85429e2f3bd7392a7274a82cee1d29ec3b3bb4af15dd6613d21e7f59cdee1405" Oct 31 19:15:27.047512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:27.047479 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r"] Oct 31 19:15:27.091270 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:27.091244 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9adf2df-9549r"] Oct 31 19:15:27.565628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:27.565596 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c" path="/var/lib/kubelet/pods/3669f8d2-9a67-4c1c-8c68-9f4b1de8a80c/volumes" Oct 31 19:15:41.528195 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:41.528157 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch"] Oct 31 19:15:54.904114 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:54.904078 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:15:56.548709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.548672 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" podUID="18556fd9-d936-40a1-9835-25216bf7f495" containerName="main" containerID="cri-o://224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07" gracePeriod=45 Oct 31 19:15:56.804628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.804568 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:15:56.814284 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.814256 5688 generic.go:356] "Generic (PLEG): container finished" podID="18556fd9-d936-40a1-9835-25216bf7f495" containerID="224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07" exitCode=0 Oct 31 19:15:56.814435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.814294 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" Oct 31 19:15:56.814435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.814297 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" event={"ID":"18556fd9-d936-40a1-9835-25216bf7f495","Type":"ContainerDied","Data":"224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07"} Oct 31 19:15:56.814435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.814348 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch" event={"ID":"18556fd9-d936-40a1-9835-25216bf7f495","Type":"ContainerDied","Data":"169e10102f5b5a700fc8c00be2995281c44549d401b1ef67982402e42774ae5d"} Oct 31 19:15:56.814435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.814371 5688 scope.go:122] "RemoveContainer" containerID="224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07" Oct 31 19:15:56.823503 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.823484 5688 scope.go:122] "RemoveContainer" containerID="224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07" Oct 31 19:15:56.823875 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:15:56.823811 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07\": container with ID starting with 224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07 not found: ID does not exist" containerID="224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07" Oct 31 19:15:56.823966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.823887 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07"} err="failed to get container status \"224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07\": rpc error: code = NotFound desc = could not find container \"224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07\": container with ID starting with 224640882ebefcd43a85ae3ef1dd2bc9ec6e57e189ab46c91a0796b31c067e07 not found: ID does not exist" Oct 31 19:15:56.912690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912661 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm\") pod \"18556fd9-d936-40a1-9835-25216bf7f495\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " Oct 31 19:15:56.912861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912703 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs\" (UniqueName: \"kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs\") pod \"18556fd9-d936-40a1-9835-25216bf7f495\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " Oct 31 19:15:56.912861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912727 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir\") pod \"18556fd9-d936-40a1-9835-25216bf7f495\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " Oct 31 19:15:56.912861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912752 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home\") pod \"18556fd9-d936-40a1-9835-25216bf7f495\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " Oct 31 19:15:56.912861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912824 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n\" (UniqueName: \"kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n\") pod \"18556fd9-d936-40a1-9835-25216bf7f495\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " Oct 31 19:15:56.913088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912959 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir" pod "18556fd9-d936-40a1-9835-25216bf7f495" (UID: "18556fd9-d936-40a1-9835-25216bf7f495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:15:56.913088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912995 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache\") pod \"18556fd9-d936-40a1-9835-25216bf7f495\" (UID: \"18556fd9-d936-40a1-9835-25216bf7f495\") " Oct 31 19:15:56.913088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.912996 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home" pod "18556fd9-d936-40a1-9835-25216bf7f495" (UID: "18556fd9-d936-40a1-9835-25216bf7f495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:15:56.913281 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.913203 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache" pod "18556fd9-d936-40a1-9835-25216bf7f495" (UID: "18556fd9-d936-40a1-9835-25216bf7f495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:15:56.913446 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.913430 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:56.913496 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.913449 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:56.913496 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.913459 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:56.915095 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.915066 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs" pod "18556fd9-d936-40a1-9835-25216bf7f495" (UID: "18556fd9-d936-40a1-9835-25216bf7f495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:15:56.915598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.915572 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm" pod "18556fd9-d936-40a1-9835-25216bf7f495" (UID: "18556fd9-d936-40a1-9835-25216bf7f495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:15:56.915675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:56.915609 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n" pod "18556fd9-d936-40a1-9835-25216bf7f495" (UID: "18556fd9-d936-40a1-9835-25216bf7f495"). InnerVolumeSpecName "kube-api-access-54h5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:15:57.014428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:57.014400 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18556fd9-d936-40a1-9835-25216bf7f495-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:57.014428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:57.014424 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18556fd9-d936-40a1-9835-25216bf7f495-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:57.014578 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:57.014434 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54h5n\" (UniqueName: \"kubernetes.io/projected/18556fd9-d936-40a1-9835-25216bf7f495-kube-api-access-54h5n\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:15:57.235791 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:57.235754 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch"] Oct 31 19:15:57.245308 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:57.245271 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-d9922f/explicit-model-d9922f-kserve-6fd7c6d887-hnbch"] Oct 31 19:15:57.565603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:15:57.565566 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18556fd9-d936-40a1-9835-25216bf7f495" path="/var/lib/kubelet/pods/18556fd9-d936-40a1-9835-25216bf7f495/volumes" Oct 31 19:16:19.946370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946336 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5"] Oct 31 19:16:19.946829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946771 5688 cpu_manager.go:414] "containerMap: removing container" podUID="18556fd9-d936-40a1-9835-25216bf7f495" containerName="main" Oct 31 19:16:19.946829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946782 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18556fd9-d936-40a1-9835-25216bf7f495" containerName="main" Oct 31 19:16:19.946829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946801 5688 cpu_manager.go:414] "containerMap: removing container" podUID="8a0da11b-3490-452c-8c1c-b346274e32c3" containerName="revoke-keys" Oct 31 19:16:19.946829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946806 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0da11b-3490-452c-8c1c-b346274e32c3" containerName="revoke-keys" Oct 31 19:16:19.946829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946821 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:16:19.946829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946826 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946834 5688 cpu_manager.go:414] "containerMap: removing container" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946839 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a93696-a204-4d0a-8a06-130b7005a245" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946850 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946856 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946868 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946873 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b6cf26-d0b2-431d-b2b3-20c7f7ceedd7" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946948 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bd8e3f-4d21-4db9-90db-283242c18ee4" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946960 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4269af6a-5192-40da-a6b1-fa77eac68fcc" containerName="cleanup" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946967 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="18556fd9-d936-40a1-9835-25216bf7f495" containerName="main" Oct 31 19:16:19.947021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.946976 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0da11b-3490-452c-8c1c-b346274e32c3" containerName="revoke-keys" Oct 31 19:16:19.950021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:19.950001 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.008312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.008283 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5"] Oct 31 19:16:20.041623 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.041593 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:16:20.049778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.049749 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:16:20.053755 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.053727 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9ced6b-dockercfg-5sjw6\"" Oct 31 19:16:20.054105 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.054086 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9ced6b\"" Oct 31 19:16:20.058994 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.058962 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwnb\" (UniqueName: \"kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb\") pod \"payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.059109 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.059068 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.098296 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.098261 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq"] Oct 31 19:16:20.102095 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.102076 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.158663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.158638 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq"] Oct 31 19:16:20.160575 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.160552 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.160703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.160680 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwnb\" (UniqueName: \"kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb\") pod \"payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.161274 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.161253 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.219987 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.219901 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwnb\" (UniqueName: \"kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb\") pod \"payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.261327 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.261301 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.261462 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.261365 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khw27\" (UniqueName: \"kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27\") pod \"payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.263341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.263321 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:20.363117 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.363026 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.363117 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.363086 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khw27\" (UniqueName: \"kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27\") pod \"payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.364789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.364758 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.420127 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.420100 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khw27\" (UniqueName: \"kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27\") pod \"payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.638701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.638678 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5"] Oct 31 19:16:20.640850 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:16:20.640822 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54e248d_d756_4fbb_82da_e4db0047b6bf.slice/crio-698b83eaa624c8b6297564c973b60c56ab1e36528edd5516bf00fd64df088c54 WatchSource:0}: Error finding container 698b83eaa624c8b6297564c973b60c56ab1e36528edd5516bf00fd64df088c54: Status 404 returned error can't find the container with id 698b83eaa624c8b6297564c973b60c56ab1e36528edd5516bf00fd64df088c54 Oct 31 19:16:20.715693 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.715667 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:20.961523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.961445 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" event={"ID":"c54e248d-d756-4fbb-82da-e4db0047b6bf","Type":"ContainerStarted","Data":"7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc"} Oct 31 19:16:20.961896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.961523 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" event={"ID":"c54e248d-d756-4fbb-82da-e4db0047b6bf","Type":"ContainerStarted","Data":"698b83eaa624c8b6297564c973b60c56ab1e36528edd5516bf00fd64df088c54"} Oct 31 19:16:20.961896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:20.961555 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:21.074972 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:21.074921 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" podStartSLOduration=2.074907741 podStartE2EDuration="2.074907741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:21.073634853 +0000 UTC m=+3301.675775997" watchObservedRunningTime="2026-10-31 19:16:21.074907741 +0000 UTC m=+3301.677048884" Oct 31 19:16:21.103285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:21.103120 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq"] Oct 31 19:16:21.106059 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:16:21.106034 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a640f64_8ecf_412c_ac97_aa245d9396fb.slice/crio-e08a89ea7877e9b6dec5346b6598f498688998d0aa1b24558c9ef9cf255db34e WatchSource:0}: Error finding container e08a89ea7877e9b6dec5346b6598f498688998d0aa1b24558c9ef9cf255db34e: Status 404 returned error can't find the container with id e08a89ea7877e9b6dec5346b6598f498688998d0aa1b24558c9ef9cf255db34e Oct 31 19:16:21.968709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:21.968670 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" event={"ID":"2a640f64-8ecf-412c-ac97-aa245d9396fb","Type":"ContainerStarted","Data":"6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b"} Oct 31 19:16:21.969110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:21.968871 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:21.969110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:21.968902 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" event={"ID":"2a640f64-8ecf-412c-ac97-aa245d9396fb","Type":"ContainerStarted","Data":"e08a89ea7877e9b6dec5346b6598f498688998d0aa1b24558c9ef9cf255db34e"} Oct 31 19:16:22.087132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:22.087080 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" podStartSLOduration=2.087067859 podStartE2EDuration="2.087067859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:22.085444072 +0000 UTC m=+3302.687585216" watchObservedRunningTime="2026-10-31 19:16:22.087067859 +0000 UTC m=+3302.689209003" Oct 31 19:16:24.903928 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:16:24.903895 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:16:31.971418 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:31.971387 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:16:32.978135 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:32.978105 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:16:54.904493 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:16:54.904457 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:16:57.957128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:57.957039 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7"] Oct 31 19:16:57.957609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:16:57.957258 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" containerID="cri-o://a706097d45e4e26d70cd8bdbf6184a97749fc139b113fb97709f6da853db2b75" gracePeriod=30 Oct 31 19:17:01.965156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:01.965118 5688 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:17:01.965561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:01.965178 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:17:03.209828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.209792 5688 generic.go:356] "Generic (PLEG): container finished" podID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerID="a706097d45e4e26d70cd8bdbf6184a97749fc139b113fb97709f6da853db2b75" exitCode=0 Oct 31 19:17:03.210155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.209857 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" event={"ID":"dce1ee45-6e69-40b9-bb09-5f02629bcd1d","Type":"ContainerDied","Data":"a706097d45e4e26d70cd8bdbf6184a97749fc139b113fb97709f6da853db2b75"} Oct 31 19:17:03.210155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.209919 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" event={"ID":"dce1ee45-6e69-40b9-bb09-5f02629bcd1d","Type":"ContainerDied","Data":"9e50c5bbf63c6ae8566ad634866e9130948cd2d999c9001d9ed6962778e7b34c"} Oct 31 19:17:03.210155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.209935 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e50c5bbf63c6ae8566ad634866e9130948cd2d999c9001d9ed6962778e7b34c" Oct 31 19:17:03.214060 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.214040 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:17:03.319301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319275 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319342 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319398 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319423 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319461 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319493 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319546 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:03.319613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319562 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:03.319613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319584 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319655 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.319913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319685 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:03.319913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319702 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:03.319913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319729 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml\") pod \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\" (UID: \"dce1ee45-6e69-40b9-bb09-5f02629bcd1d\") " Oct 31 19:17:03.320120 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.319970 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:03.320266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.320248 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.320343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.320272 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.320343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.320304 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.320343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.320320 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.320343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.320332 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.322102 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.322072 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "kube-api-access-lksml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:03.322228 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.322102 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:03.322309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.322292 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:17:03.322362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.322310 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token" pod "dce1ee45-6e69-40b9-bb09-5f02629bcd1d" (UID: "dce1ee45-6e69-40b9-bb09-5f02629bcd1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:03.421707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.421678 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.421707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.421702 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.421707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.421712 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:03.421989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:03.421722 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lksml\" (UniqueName: \"kubernetes.io/projected/dce1ee45-6e69-40b9-bb09-5f02629bcd1d-kube-api-access-lksml\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:04.217676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:04.217641 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7" Oct 31 19:17:04.430456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:04.430422 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7"] Oct 31 19:17:04.480267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:04.480175 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-mvqk7"] Oct 31 19:17:05.566512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:05.566485 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" path="/var/lib/kubelet/pods/dce1ee45-6e69-40b9-bb09-5f02629bcd1d/volumes" Oct 31 19:17:21.313529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:21.313491 5688 scope.go:122] "RemoveContainer" containerID="433d37be898dc6ee031e98b6adf134e046a8854bff98b5cf2363a73e1b9638fe" Oct 31 19:17:21.323557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:21.323517 5688 scope.go:122] "RemoveContainer" containerID="15458262f070e277ea11a79eb51ed33d1e6a3769c84fda24abb620faa53560e8" Oct 31 19:17:21.333096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:21.333075 5688 scope.go:122] "RemoveContainer" containerID="a706097d45e4e26d70cd8bdbf6184a97749fc139b113fb97709f6da853db2b75" Oct 31 19:17:24.905105 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:17:24.905073 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:17:26.374652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.374620 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp"] Oct 31 19:17:26.375241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.375223 5688 cpu_manager.go:414] "containerMap: removing container" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" Oct 31 19:17:26.375312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.375242 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" Oct 31 19:17:26.375360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.375350 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce1ee45-6e69-40b9-bb09-5f02629bcd1d" containerName="istio-proxy" Oct 31 19:17:26.380047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.380029 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.436311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.436286 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp"] Oct 31 19:17:26.482915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.482884 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:17:26.483484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.483335 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:17:26.551050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.551016 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprsp\" (UniqueName: \"kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.551255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.551068 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.551337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.551286 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.652564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.652481 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.652732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.652619 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.652732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.652709 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprsp\" (UniqueName: \"kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.653321 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.653177 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.655197 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.655155 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.710998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.710971 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprsp\" (UniqueName: \"kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp\") pod \"maas-api-revoke-sub-d6a163ff-v86pp\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:26.993445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:26.993421 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:27.367780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:27.367755 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp"] Oct 31 19:17:27.369814 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:17:27.369789 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a4fbd_efbe_43d0_aefe_ed798d04d3df.slice/crio-16a9fbb8e8205d1bad4c781e09978c5b2b07e8f0f5bd30df882162b1d54a869b WatchSource:0}: Error finding container 16a9fbb8e8205d1bad4c781e09978c5b2b07e8f0f5bd30df882162b1d54a869b: Status 404 returned error can't find the container with id 16a9fbb8e8205d1bad4c781e09978c5b2b07e8f0f5bd30df882162b1d54a869b Oct 31 19:17:28.362291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:28.362257 5688 generic.go:356] "Generic (PLEG): container finished" podID="ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" containerID="7d3a122a042a45cb11c70553d1133567462cc5f24ccd512fb9edc967a2b2c8ee" exitCode=0 Oct 31 19:17:28.362658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:28.362346 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" event={"ID":"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df","Type":"ContainerDied","Data":"7d3a122a042a45cb11c70553d1133567462cc5f24ccd512fb9edc967a2b2c8ee"} Oct 31 19:17:28.362658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:28.362394 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" event={"ID":"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df","Type":"ContainerStarted","Data":"16a9fbb8e8205d1bad4c781e09978c5b2b07e8f0f5bd30df882162b1d54a869b"} Oct 31 19:17:29.521557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.521535 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:29.684206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.684106 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp\" (UniqueName: \"kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp\") pod \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " Oct 31 19:17:29.684389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.684220 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca\") pod \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " Oct 31 19:17:29.684389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.684328 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca\") pod \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\" (UID: \"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df\") " Oct 31 19:17:29.684682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.684649 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca" pod "ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" (UID: "ce9a4fbd-efbe-43d0-aefe-ed798d04d3df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:29.686516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.686493 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp" pod "ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" (UID: "ce9a4fbd-efbe-43d0-aefe-ed798d04d3df"). InnerVolumeSpecName "kube-api-access-tprsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:29.686516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.686493 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca" pod "ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" (UID: "ce9a4fbd-efbe-43d0-aefe-ed798d04d3df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:29.785819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.785784 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:29.785819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.785815 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:29.785988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:29.785830 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprsp\" (UniqueName: \"kubernetes.io/projected/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df-kube-api-access-tprsp\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:30.382548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:30.382514 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" Oct 31 19:17:30.382705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:30.382517 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp" event={"ID":"ce9a4fbd-efbe-43d0-aefe-ed798d04d3df","Type":"ContainerDied","Data":"16a9fbb8e8205d1bad4c781e09978c5b2b07e8f0f5bd30df882162b1d54a869b"} Oct 31 19:17:30.382705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:30.382654 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16a9fbb8e8205d1bad4c781e09978c5b2b07e8f0f5bd30df882162b1d54a869b" Oct 31 19:17:37.119968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.119934 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz"] Oct 31 19:17:37.120443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.120429 5688 cpu_manager.go:414] "containerMap: removing container" podUID="ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" containerName="revoke-keys" Oct 31 19:17:37.120488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.120443 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" containerName="revoke-keys" Oct 31 19:17:37.120523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.120517 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" containerName="revoke-keys" Oct 31 19:17:37.123920 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.123904 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.154100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.154074 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.154245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.154112 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.154245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.154140 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4r4\" (UniqueName: \"kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.180697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.180671 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz"] Oct 31 19:17:37.188745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.188726 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:17:37.227080 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.227062 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:17:37.255319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.255294 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.255432 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.255352 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.255432 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.255407 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4r4\" (UniqueName: \"kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.255908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.255890 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.257701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.257676 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.311657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.311632 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4r4\" (UniqueName: \"kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4\") pod \"maas-api-revoke-sub-414a52fd-z2bkz\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.436853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.436793 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:37.812165 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:37.812131 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz"] Oct 31 19:17:37.814494 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:17:37.814464 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd674ae_b74b_45fb_a95f_fd5333092a28.slice/crio-4aa16d2f4aab7859166005387f7e8d10d952b420efc82e20a44b332743171540 WatchSource:0}: Error finding container 4aa16d2f4aab7859166005387f7e8d10d952b420efc82e20a44b332743171540: Status 404 returned error can't find the container with id 4aa16d2f4aab7859166005387f7e8d10d952b420efc82e20a44b332743171540 Oct 31 19:17:38.428313 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:38.428235 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" event={"ID":"dcd674ae-b74b-45fb-a95f-fd5333092a28","Type":"ContainerStarted","Data":"79a61a4b6943474e5f0995d9e3a35447e8baa41490ce2be255117b7e11fe0d8d"} Oct 31 19:17:38.428313 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:38.428286 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" event={"ID":"dcd674ae-b74b-45fb-a95f-fd5333092a28","Type":"ContainerStarted","Data":"4aa16d2f4aab7859166005387f7e8d10d952b420efc82e20a44b332743171540"} Oct 31 19:17:38.546009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:38.545920 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" podStartSLOduration=1.545903931 podStartE2EDuration="1.545903931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:38.545432736 +0000 UTC m=+3379.147573858" watchObservedRunningTime="2026-10-31 19:17:38.545903931 +0000 UTC m=+3379.148045076" Oct 31 19:17:39.435486 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:39.435443 5688 generic.go:356] "Generic (PLEG): container finished" podID="dcd674ae-b74b-45fb-a95f-fd5333092a28" containerID="79a61a4b6943474e5f0995d9e3a35447e8baa41490ce2be255117b7e11fe0d8d" exitCode=0 Oct 31 19:17:39.435864 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:39.435526 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" event={"ID":"dcd674ae-b74b-45fb-a95f-fd5333092a28","Type":"ContainerDied","Data":"79a61a4b6943474e5f0995d9e3a35447e8baa41490ce2be255117b7e11fe0d8d"} Oct 31 19:17:40.588599 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.588578 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:40.684095 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.684068 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4\" (UniqueName: \"kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4\") pod \"dcd674ae-b74b-45fb-a95f-fd5333092a28\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " Oct 31 19:17:40.684226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.684124 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca\") pod \"dcd674ae-b74b-45fb-a95f-fd5333092a28\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " Oct 31 19:17:40.686285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.686258 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca" pod "dcd674ae-b74b-45fb-a95f-fd5333092a28" (UID: "dcd674ae-b74b-45fb-a95f-fd5333092a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:40.686285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.686269 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4" pod "dcd674ae-b74b-45fb-a95f-fd5333092a28" (UID: "dcd674ae-b74b-45fb-a95f-fd5333092a28"). InnerVolumeSpecName "kube-api-access-xm4r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:40.785417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.785383 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca\") pod \"dcd674ae-b74b-45fb-a95f-fd5333092a28\" (UID: \"dcd674ae-b74b-45fb-a95f-fd5333092a28\") " Oct 31 19:17:40.785678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.785659 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca" pod "dcd674ae-b74b-45fb-a95f-fd5333092a28" (UID: "dcd674ae-b74b-45fb-a95f-fd5333092a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:40.785822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.785808 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:40.785855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.785828 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd674ae-b74b-45fb-a95f-fd5333092a28-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:40.785855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:40.785845 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4r4\" (UniqueName: \"kubernetes.io/projected/dcd674ae-b74b-45fb-a95f-fd5333092a28-kube-api-access-xm4r4\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:17:41.447619 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:41.447592 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" Oct 31 19:17:41.447619 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:41.447603 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz" event={"ID":"dcd674ae-b74b-45fb-a95f-fd5333092a28","Type":"ContainerDied","Data":"4aa16d2f4aab7859166005387f7e8d10d952b420efc82e20a44b332743171540"} Oct 31 19:17:41.447813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:17:41.447655 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa16d2f4aab7859166005387f7e8d10d952b420efc82e20a44b332743171540" Oct 31 19:17:54.904425 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:17:54.904387 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:18:24.903595 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:18:24.903563 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:18:28.639756 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.639682 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx"] Oct 31 19:18:28.640240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.640224 5688 cpu_manager.go:414] "containerMap: removing container" podUID="dcd674ae-b74b-45fb-a95f-fd5333092a28" containerName="revoke-keys" Oct 31 19:18:28.640292 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.640241 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd674ae-b74b-45fb-a95f-fd5333092a28" containerName="revoke-keys" Oct 31 19:18:28.640344 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.640334 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd674ae-b74b-45fb-a95f-fd5333092a28" containerName="revoke-keys" Oct 31 19:18:28.643444 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.643430 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.644681 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.644649 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.644825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.644807 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bx97\" (UniqueName: \"kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.644898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.644851 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.739625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.739600 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:18:28.739625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.739616 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:18:28.745552 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.745529 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.745634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.745560 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bx97\" (UniqueName: \"kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.745634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.745586 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.746200 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.746163 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.747981 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.747951 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.748597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.748578 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx"] Oct 31 19:18:28.804107 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.804082 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bx97\" (UniqueName: \"kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97\") pod \"maas-api-revoke-sub-8eb7b641-grbrx\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:28.956087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:28.956030 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:29.334831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:29.334795 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx"] Oct 31 19:18:29.337855 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:18:29.337827 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf35085c_eb02_4963_8f12_b50474554c9a.slice/crio-3c6b6845947167a04f6765318e4138e9ec9ac98b0899fc60854ffdfbd91a39f3 WatchSource:0}: Error finding container 3c6b6845947167a04f6765318e4138e9ec9ac98b0899fc60854ffdfbd91a39f3: Status 404 returned error can't find the container with id 3c6b6845947167a04f6765318e4138e9ec9ac98b0899fc60854ffdfbd91a39f3 Oct 31 19:18:29.710673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:29.710641 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" event={"ID":"bf35085c-eb02-4963-8f12-b50474554c9a","Type":"ContainerStarted","Data":"02586bd1b4cfcf7525052b1b735d8d9d6bff37fa08388c1004693f1efad32cc9"} Oct 31 19:18:29.710992 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:29.710700 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" event={"ID":"bf35085c-eb02-4963-8f12-b50474554c9a","Type":"ContainerStarted","Data":"3c6b6845947167a04f6765318e4138e9ec9ac98b0899fc60854ffdfbd91a39f3"} Oct 31 19:18:29.837357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:29.837305 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" podStartSLOduration=1.837287344 podStartE2EDuration="1.837287344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:29.836437948 +0000 UTC m=+3430.438579092" watchObservedRunningTime="2026-10-31 19:18:29.837287344 +0000 UTC m=+3430.439428488" Oct 31 19:18:30.717971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:30.717936 5688 generic.go:356] "Generic (PLEG): container finished" podID="bf35085c-eb02-4963-8f12-b50474554c9a" containerID="02586bd1b4cfcf7525052b1b735d8d9d6bff37fa08388c1004693f1efad32cc9" exitCode=0 Oct 31 19:18:30.718428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:30.717996 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" event={"ID":"bf35085c-eb02-4963-8f12-b50474554c9a","Type":"ContainerDied","Data":"02586bd1b4cfcf7525052b1b735d8d9d6bff37fa08388c1004693f1efad32cc9"} Oct 31 19:18:31.868020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.867993 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:31.988923 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.988897 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97\" (UniqueName: \"kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97\") pod \"bf35085c-eb02-4963-8f12-b50474554c9a\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " Oct 31 19:18:31.989071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.988928 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca\") pod \"bf35085c-eb02-4963-8f12-b50474554c9a\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " Oct 31 19:18:31.989071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.988994 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca\") pod \"bf35085c-eb02-4963-8f12-b50474554c9a\" (UID: \"bf35085c-eb02-4963-8f12-b50474554c9a\") " Oct 31 19:18:31.989400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.989376 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca" pod "bf35085c-eb02-4963-8f12-b50474554c9a" (UID: "bf35085c-eb02-4963-8f12-b50474554c9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:18:31.991093 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.991071 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97" pod "bf35085c-eb02-4963-8f12-b50474554c9a" (UID: "bf35085c-eb02-4963-8f12-b50474554c9a"). InnerVolumeSpecName "kube-api-access-7bx97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:18:31.991202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:31.991120 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca" pod "bf35085c-eb02-4963-8f12-b50474554c9a" (UID: "bf35085c-eb02-4963-8f12-b50474554c9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:18:32.090513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:32.090485 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bx97\" (UniqueName: \"kubernetes.io/projected/bf35085c-eb02-4963-8f12-b50474554c9a-kube-api-access-7bx97\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:18:32.090513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:32.090513 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:18:32.090705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:32.090523 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf35085c-eb02-4963-8f12-b50474554c9a-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:18:32.730766 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:32.730741 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" Oct 31 19:18:32.730942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:32.730778 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx" event={"ID":"bf35085c-eb02-4963-8f12-b50474554c9a","Type":"ContainerDied","Data":"3c6b6845947167a04f6765318e4138e9ec9ac98b0899fc60854ffdfbd91a39f3"} Oct 31 19:18:32.730942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:32.730820 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c6b6845947167a04f6765318e4138e9ec9ac98b0899fc60854ffdfbd91a39f3" Oct 31 19:18:39.417197 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.417152 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz"] Oct 31 19:18:39.417652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.417638 5688 cpu_manager.go:414] "containerMap: removing container" podUID="bf35085c-eb02-4963-8f12-b50474554c9a" containerName="revoke-keys" Oct 31 19:18:39.417694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.417653 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf35085c-eb02-4963-8f12-b50474554c9a" containerName="revoke-keys" Oct 31 19:18:39.417744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.417736 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf35085c-eb02-4963-8f12-b50474554c9a" containerName="revoke-keys" Oct 31 19:18:39.421219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.421204 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.460980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.460956 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.461091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.460996 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt4k\" (UniqueName: \"kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.461132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.461108 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.479523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.479485 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:18:39.525336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.525316 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:18:39.527690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.527668 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz"] Oct 31 19:18:39.562853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.562819 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.563134 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.563114 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.563896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.563871 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt4k\" (UniqueName: \"kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.564251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.564167 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.566870 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.566441 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.622474 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.622448 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt4k\" (UniqueName: \"kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k\") pod \"maas-api-revoke-sub-1d96a1c7-4sscz\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:39.734165 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:39.734140 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:40.123388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:40.123355 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz"] Oct 31 19:18:40.124218 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:18:40.124169 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf2d8e4_34b2_4992_8ff8_d8d646e2ec70.slice/crio-e41d962cc7f2296575dc45edbb71edebb81522542be5bbaf42e938413e74f052 WatchSource:0}: Error finding container e41d962cc7f2296575dc45edbb71edebb81522542be5bbaf42e938413e74f052: Status 404 returned error can't find the container with id e41d962cc7f2296575dc45edbb71edebb81522542be5bbaf42e938413e74f052 Oct 31 19:18:40.775502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:40.775446 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" event={"ID":"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70","Type":"ContainerStarted","Data":"81e6010916e904e09da35c2c6e265ad881d232f6f3a0e807a51a50683a0bb2ec"} Oct 31 19:18:40.776009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:40.775528 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" event={"ID":"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70","Type":"ContainerStarted","Data":"e41d962cc7f2296575dc45edbb71edebb81522542be5bbaf42e938413e74f052"} Oct 31 19:18:40.895692 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:40.895638 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" podStartSLOduration=1.895624283 podStartE2EDuration="1.895624283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:40.894459429 +0000 UTC m=+3441.496600572" watchObservedRunningTime="2026-10-31 19:18:40.895624283 +0000 UTC m=+3441.497765426" Oct 31 19:18:41.782740 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:41.782703 5688 generic.go:356] "Generic (PLEG): container finished" podID="adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" containerID="81e6010916e904e09da35c2c6e265ad881d232f6f3a0e807a51a50683a0bb2ec" exitCode=0 Oct 31 19:18:41.783238 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:41.782789 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" event={"ID":"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70","Type":"ContainerDied","Data":"81e6010916e904e09da35c2c6e265ad881d232f6f3a0e807a51a50683a0bb2ec"} Oct 31 19:18:42.931853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:42.931819 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:43.000990 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.000960 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k\" (UniqueName: \"kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k\") pod \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " Oct 31 19:18:43.001174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.001001 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca\") pod \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " Oct 31 19:18:43.001174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.001021 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca\") pod \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\" (UID: \"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70\") " Oct 31 19:18:43.001417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.001397 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca" pod "adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" (UID: "adf2d8e4-34b2-4992-8ff8-d8d646e2ec70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:18:43.003110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.003086 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca" pod "adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" (UID: "adf2d8e4-34b2-4992-8ff8-d8d646e2ec70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:18:43.003259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.003239 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k" pod "adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" (UID: "adf2d8e4-34b2-4992-8ff8-d8d646e2ec70"). InnerVolumeSpecName "kube-api-access-ptt4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:18:43.101917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.101853 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt4k\" (UniqueName: \"kubernetes.io/projected/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-kube-api-access-ptt4k\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:18:43.101917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.101880 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:18:43.101917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.101889 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:18:43.798108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.798077 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" Oct 31 19:18:43.798314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.798078 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz" event={"ID":"adf2d8e4-34b2-4992-8ff8-d8d646e2ec70","Type":"ContainerDied","Data":"e41d962cc7f2296575dc45edbb71edebb81522542be5bbaf42e938413e74f052"} Oct 31 19:18:43.798314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:43.798243 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41d962cc7f2296575dc45edbb71edebb81522542be5bbaf42e938413e74f052" Oct 31 19:18:52.453721 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.453684 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7"] Oct 31 19:18:52.454170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.454156 5688 cpu_manager.go:414] "containerMap: removing container" podUID="adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" containerName="revoke-keys" Oct 31 19:18:52.454287 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.454172 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" containerName="revoke-keys" Oct 31 19:18:52.454341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.454320 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" containerName="revoke-keys" Oct 31 19:18:52.457380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.457363 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.512152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.512131 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9cf479\"" Oct 31 19:18:52.561774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.561736 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9cf479-dockercfg-ztk8v\"" Oct 31 19:18:52.564172 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.564144 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7"] Oct 31 19:18:52.567231 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.567202 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs"] Oct 31 19:18:52.571553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.571156 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.606223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.606175 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtt6\" (UniqueName: \"kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6\") pod \"payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.606379 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.606292 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.647865 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.647834 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs"] Oct 31 19:18:52.707209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.707130 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fs5n\" (UniqueName: \"kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n\") pod \"payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.707209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.707173 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.707426 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.707399 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtt6\" (UniqueName: \"kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6\") pod \"payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.707484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.707466 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.707953 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.707935 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.767268 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.767249 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtt6\" (UniqueName: \"kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6\") pod \"payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.771201 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.771153 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:52.809334 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.809306 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.809481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.809373 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fs5n\" (UniqueName: \"kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n\") pod \"payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.809961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.809939 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.872866 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.872837 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fs5n\" (UniqueName: \"kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n\") pod \"payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:52.897658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:52.897633 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:53.154003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.153968 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7"] Oct 31 19:18:53.156944 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:18:53.156913 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c53ca5_64e9_46be_ab0b_c68dcc50a4fd.slice/crio-5b5caa18e964e9177de12bcfe1334711a6474ba003e44938000e40092bcdc499 WatchSource:0}: Error finding container 5b5caa18e964e9177de12bcfe1334711a6474ba003e44938000e40092bcdc499: Status 404 returned error can't find the container with id 5b5caa18e964e9177de12bcfe1334711a6474ba003e44938000e40092bcdc499 Oct 31 19:18:53.296394 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:18:53.296363 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17c2cff_4777_4943_8bf7_38e1d5c07d72.slice/crio-4e9bf2ad803faee9716efc0ad2de958d1ad5f45965c7c2a65b45c21815bce1a2 WatchSource:0}: Error finding container 4e9bf2ad803faee9716efc0ad2de958d1ad5f45965c7c2a65b45c21815bce1a2: Status 404 returned error can't find the container with id 4e9bf2ad803faee9716efc0ad2de958d1ad5f45965c7c2a65b45c21815bce1a2 Oct 31 19:18:53.298887 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.298865 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs"] Oct 31 19:18:53.859706 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.859662 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" event={"ID":"c17c2cff-4777-4943-8bf7-38e1d5c07d72","Type":"ContainerStarted","Data":"141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8"} Oct 31 19:18:53.860148 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.859730 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" event={"ID":"c17c2cff-4777-4943-8bf7-38e1d5c07d72","Type":"ContainerStarted","Data":"4e9bf2ad803faee9716efc0ad2de958d1ad5f45965c7c2a65b45c21815bce1a2"} Oct 31 19:18:53.860148 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.859760 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:18:53.861551 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.861524 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" event={"ID":"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd","Type":"ContainerStarted","Data":"654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150"} Oct 31 19:18:53.861709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.861565 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" event={"ID":"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd","Type":"ContainerStarted","Data":"5b5caa18e964e9177de12bcfe1334711a6474ba003e44938000e40092bcdc499"} Oct 31 19:18:53.861709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.861621 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:18:53.984807 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:18:53.984755 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" podStartSLOduration=1.98474273 podStartE2EDuration="1.98474273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:53.982807594 +0000 UTC m=+3454.584948738" watchObservedRunningTime="2026-10-31 19:18:53.98474273 +0000 UTC m=+3454.586883873" Oct 31 19:18:54.906852 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:18:54.906816 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:19:04.869726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:04.869698 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:19:04.870239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:04.869768 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:19:04.998207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:04.998121 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" podStartSLOduration=12.998106495 podStartE2EDuration="12.998106495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:54.107803008 +0000 UTC m=+3454.709944153" watchObservedRunningTime="2026-10-31 19:19:04.998106495 +0000 UTC m=+3465.600247679" Oct 31 19:19:06.972100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:06.972066 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp"] Oct 31 19:19:06.977005 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:06.976989 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.030285 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.030265 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-8103d4\"" Oct 31 19:19:07.064132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.063656 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhwvk\" (UniqueName: \"kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk\") pod \"payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.064132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.063810 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.084485 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.084450 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-8103d4-dockercfg-mlnwr\"" Oct 31 19:19:07.087647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.087619 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp"] Oct 31 19:19:07.131853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.131807 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg"] Oct 31 19:19:07.136029 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.136007 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.164529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.164503 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2scm\" (UniqueName: \"kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm\") pod \"payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.164654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.164543 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhwvk\" (UniqueName: \"kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk\") pod \"payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.164654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.164626 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.164800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.164785 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.165490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.165455 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.175725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.175702 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg"] Oct 31 19:19:07.229611 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.229538 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhwvk\" (UniqueName: \"kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk\") pod \"payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.265963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.265929 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2scm\" (UniqueName: \"kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm\") pod \"payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.266154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.265985 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.266800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.266771 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.292565 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.292538 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.329471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.329442 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2scm\" (UniqueName: \"kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm\") pod \"payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.449204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.449151 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:07.689512 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.689486 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp"] Oct 31 19:19:07.690425 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:19:07.690394 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420222b8_86cc_4534_8323_7a6092b6e09f.slice/crio-3942ff8512b27cffad3f82b3d8338438c7f5f0e08f824fb77924aac1f672869d WatchSource:0}: Error finding container 3942ff8512b27cffad3f82b3d8338438c7f5f0e08f824fb77924aac1f672869d: Status 404 returned error can't find the container with id 3942ff8512b27cffad3f82b3d8338438c7f5f0e08f824fb77924aac1f672869d Oct 31 19:19:07.852858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.852744 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg"] Oct 31 19:19:07.950604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.950570 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" event={"ID":"420222b8-86cc-4534-8323-7a6092b6e09f","Type":"ContainerStarted","Data":"351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b"} Oct 31 19:19:07.950751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.950634 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" event={"ID":"420222b8-86cc-4534-8323-7a6092b6e09f","Type":"ContainerStarted","Data":"3942ff8512b27cffad3f82b3d8338438c7f5f0e08f824fb77924aac1f672869d"} Oct 31 19:19:07.950751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.950665 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:07.952158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.952135 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" event={"ID":"5276e29e-70b6-42e6-bdf2-f4f859ceaf08","Type":"ContainerStarted","Data":"908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b"} Oct 31 19:19:07.952278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.952178 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" event={"ID":"5276e29e-70b6-42e6-bdf2-f4f859ceaf08","Type":"ContainerStarted","Data":"18f882abfaa56f8da7fc8beb2ea12373ffd6cdec47eafb5a7bb3c1876e91bc4a"} Oct 31 19:19:07.952278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:07.952249 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:08.079606 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:08.079527 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" podStartSLOduration=2.079512374 podStartE2EDuration="2.079512374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:08.079213724 +0000 UTC m=+3468.681354865" watchObservedRunningTime="2026-10-31 19:19:08.079512374 +0000 UTC m=+3468.681653521" Oct 31 19:19:08.222230 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:08.222152 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" podStartSLOduration=1.2221335070000001 podStartE2EDuration="1.222133507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:08.211603067 +0000 UTC m=+3468.813744212" watchObservedRunningTime="2026-10-31 19:19:08.222133507 +0000 UTC m=+3468.824274651" Oct 31 19:19:11.937517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:11.937482 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w"] Oct 31 19:19:11.941580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:11.941562 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:11.994392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:11.994373 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-150680-dockercfg-cbvn7\"" Oct 31 19:19:12.015391 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.015362 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626ks\" (UniqueName: \"kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks\") pod \"payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.015517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.015413 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.048948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.048917 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-150680\"" Oct 31 19:19:12.051771 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.051743 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w"] Oct 31 19:19:12.110937 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.110905 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x"] Oct 31 19:19:12.115178 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.115155 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.116056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.116026 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626ks\" (UniqueName: \"kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks\") pod \"payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.116152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.116074 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.116819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.116795 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.176388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.176361 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626ks\" (UniqueName: \"kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks\") pod \"payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.183256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.183229 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x"] Oct 31 19:19:12.217484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.217416 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.217626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.217606 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qmxn\" (UniqueName: \"kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn\") pod \"payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.255817 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.255790 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.319282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.319101 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.319428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.319336 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qmxn\" (UniqueName: \"kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn\") pod \"payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.321283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.321253 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.377554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.377521 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qmxn\" (UniqueName: \"kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn\") pod \"payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.429934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.429912 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:12.638719 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.638694 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w"] Oct 31 19:19:12.639594 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:19:12.639567 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154938fe_a662_4607_9f25_197e025f7953.slice/crio-49151b4dd979f123e4340f1287f49097ef0db7c93bdc6c14e52db4e01b78ce46 WatchSource:0}: Error finding container 49151b4dd979f123e4340f1287f49097ef0db7c93bdc6c14e52db4e01b78ce46: Status 404 returned error can't find the container with id 49151b4dd979f123e4340f1287f49097ef0db7c93bdc6c14e52db4e01b78ce46 Oct 31 19:19:12.816411 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.816389 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x"] Oct 31 19:19:12.817153 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:19:12.817131 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457a6bd6_dfb3_4e0e_b81f_5ba6c295fe0b.slice/crio-75aecd8d4dc3837e53fc29b38db35eeca084b6ff8d80f28f46c58c9980b0cce7 WatchSource:0}: Error finding container 75aecd8d4dc3837e53fc29b38db35eeca084b6ff8d80f28f46c58c9980b0cce7: Status 404 returned error can't find the container with id 75aecd8d4dc3837e53fc29b38db35eeca084b6ff8d80f28f46c58c9980b0cce7 Oct 31 19:19:12.982017 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.981986 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" event={"ID":"154938fe-a662-4607-9f25-197e025f7953","Type":"ContainerStarted","Data":"72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9"} Oct 31 19:19:12.982418 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.982043 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" event={"ID":"154938fe-a662-4607-9f25-197e025f7953","Type":"ContainerStarted","Data":"49151b4dd979f123e4340f1287f49097ef0db7c93bdc6c14e52db4e01b78ce46"} Oct 31 19:19:12.982418 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.982073 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:12.983610 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.983589 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" event={"ID":"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b","Type":"ContainerStarted","Data":"87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3"} Oct 31 19:19:12.983661 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.983629 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" event={"ID":"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b","Type":"ContainerStarted","Data":"75aecd8d4dc3837e53fc29b38db35eeca084b6ff8d80f28f46c58c9980b0cce7"} Oct 31 19:19:12.983695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:12.983657 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:13.104150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:13.104062 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" podStartSLOduration=2.104049351 podStartE2EDuration="2.104049351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:13.103545646 +0000 UTC m=+3473.705686802" watchObservedRunningTime="2026-10-31 19:19:13.104049351 +0000 UTC m=+3473.706190494" Oct 31 19:19:13.235207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:13.231991 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" podStartSLOduration=1.231971814 podStartE2EDuration="1.231971814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:13.229410448 +0000 UTC m=+3473.831551593" watchObservedRunningTime="2026-10-31 19:19:13.231971814 +0000 UTC m=+3473.834112965" Oct 31 19:19:18.962031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:18.961999 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:19:18.962482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:18.962077 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:19:23.992561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:23.992532 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:19:23.992934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:23.992746 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:19:24.904503 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:19:24.904468 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:19:29.053712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:29.053679 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg"] Oct 31 19:19:29.104272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:29.104224 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22173abe-4t5wg"] Oct 31 19:19:29.575832 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:29.575794 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b12386-78be-41f8-a3a4-13e355da0ef8" path="/var/lib/kubelet/pods/51b12386-78be-41f8-a3a4-13e355da0ef8/volumes" Oct 31 19:19:40.064690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:40.064653 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2"] Oct 31 19:19:40.114352 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:40.114316 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-860ea8a9-fd7s2"] Oct 31 19:19:40.224813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:40.224780 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl"] Oct 31 19:19:41.567332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:41.567296 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe95fb3-e815-42fe-a591-aeee67dce92f" path="/var/lib/kubelet/pods/ebe95fb3-e815-42fe-a591-aeee67dce92f/volumes" Oct 31 19:19:54.904904 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:19:54.904871 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:19:55.245497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.245462 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" podUID="81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" containerName="main" containerID="cri-o://4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c" gracePeriod=45 Oct 31 19:19:55.509096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.509043 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:19:55.653533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653504 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm\") pod \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " Oct 31 19:19:55.653700 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653581 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir\") pod \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " Oct 31 19:19:55.653700 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653610 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache\") pod \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " Oct 31 19:19:55.653787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653745 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2\" (UniqueName: \"kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2\") pod \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " Oct 31 19:19:55.653787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653778 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home\") pod \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " Oct 31 19:19:55.653865 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653822 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir" pod "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" (UID: "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:19:55.653865 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653834 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs\" (UniqueName: \"kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs\") pod \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\" (UID: \"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc\") " Oct 31 19:19:55.653967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.653899 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache" pod "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" (UID: "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:19:55.654160 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.654125 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home" pod "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" (UID: "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:19:55.654312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.654295 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:19:55.654387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.654315 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:19:55.654387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.654328 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:19:55.656477 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.656451 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm" pod "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" (UID: "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:19:55.656595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.656568 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs" pod "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" (UID: "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:19:55.656595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.656575 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2" pod "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" (UID: "81ed5b92-f253-47d7-98be-ffb3d4fdfcdc"). InnerVolumeSpecName "kube-api-access-trxm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:19:55.755650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.755621 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trxm2\" (UniqueName: \"kubernetes.io/projected/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-kube-api-access-trxm2\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:19:55.755650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.755642 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:19:55.755650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:55.755650 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:19:56.278902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.278872 5688 generic.go:356] "Generic (PLEG): container finished" podID="81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" containerID="4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c" exitCode=0 Oct 31 19:19:56.279362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.278926 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" Oct 31 19:19:56.279362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.278932 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" event={"ID":"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc","Type":"ContainerDied","Data":"4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c"} Oct 31 19:19:56.279362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.279034 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl" event={"ID":"81ed5b92-f253-47d7-98be-ffb3d4fdfcdc","Type":"ContainerDied","Data":"cd20a2c9f2a23532c202cba93066efc07e1d34ad19a7ebd7110530d3d663919f"} Oct 31 19:19:56.279362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.279049 5688 scope.go:122] "RemoveContainer" containerID="4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c" Oct 31 19:19:56.289632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.289615 5688 scope.go:122] "RemoveContainer" containerID="4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c" Oct 31 19:19:56.289897 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:19:56.289876 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c\": container with ID starting with 4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c not found: ID does not exist" containerID="4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c" Oct 31 19:19:56.289945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.289905 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c"} err="failed to get container status \"4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c\": rpc error: code = NotFound desc = could not find container \"4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c\": container with ID starting with 4c0ebb2ed0311044dcffe4e312e2ee910039a93ab93c9621c3293659359a188c not found: ID does not exist" Oct 31 19:19:56.443166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.443141 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl"] Oct 31 19:19:56.447087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:56.447066 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/auth-test-model-1aefcd-kserve-6c775946b9-m7ssl"] Oct 31 19:19:57.566547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:19:57.566515 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" path="/var/lib/kubelet/pods/81ed5b92-f253-47d7-98be-ffb3d4fdfcdc/volumes" Oct 31 19:20:08.006011 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.005969 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh"] Oct 31 19:20:08.006689 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.006668 5688 cpu_manager.go:414] "containerMap: removing container" podUID="81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" containerName="main" Oct 31 19:20:08.006797 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.006691 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" containerName="main" Oct 31 19:20:08.006888 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.006875 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ed5b92-f253-47d7-98be-ffb3d4fdfcdc" containerName="main" Oct 31 19:20:08.010252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.010231 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.096837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.096799 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.097023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.096865 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.097023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.096960 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mvj\" (UniqueName: \"kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.097023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.097008 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.115360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.115322 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9cf479-serving-cert\"" Oct 31 19:20:08.117860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.117839 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh"] Oct 31 19:20:08.198354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.198322 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.198506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.198367 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.198506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.198405 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mvj\" (UniqueName: \"kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.198506 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.198436 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.198623 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:20:08.198563 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-serving-cert: secret "maas-api-e2e-extmod-a-9cf479-serving-cert" not found Oct 31 19:20:08.198681 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:20:08.198650 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls podName:1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c nodeName:}" failed. No retries permitted until 2026-10-31 19:20:08.69862695 +0000 UTC m=+3529.300768074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls") pod "maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" (UID: "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c") : secret "maas-api-e2e-extmod-a-9cf479-serving-cert" not found Oct 31 19:20:08.199124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.199102 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.200834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.200815 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.257383 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.257282 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mvj\" (UniqueName: \"kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.703951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.703870 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.706404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.706385 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:08.923651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:08.923621 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:09.300266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:09.300240 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh"] Oct 31 19:20:09.306554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:09.306534 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:20:09.359897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:09.359866 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" event={"ID":"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c","Type":"ContainerStarted","Data":"9d28a00c548106923dddf3f021b37e9bedd6bac110cb9a82376fbf2756f768fb"} Oct 31 19:20:10.367686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:10.367651 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" event={"ID":"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c","Type":"ContainerStarted","Data":"183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64"} Oct 31 19:20:10.368070 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:10.367737 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:10.501131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:10.501083 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" podStartSLOduration=2.60671432 podStartE2EDuration="3.501070211s" totalImagesPullingTime="894.355891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:10.499294373 +0000 UTC m=+3531.101435518" watchObservedRunningTime="2026-10-31 19:20:10.501070211 +0000 UTC m=+3531.103211380" Oct 31 19:20:14.049898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:14.049863 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n"] Oct 31 19:20:14.099421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:14.099394 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a50397-8w67n"] Oct 31 19:20:15.566965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:15.566933 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d735f3-d157-4ef6-8aeb-5f9781f2b2c8" path="/var/lib/kubelet/pods/22d735f3-d157-4ef6-8aeb-5f9781f2b2c8/volumes" Oct 31 19:20:16.378406 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:16.378378 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:20:24.904909 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:20:24.904870 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:20:25.046569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:25.046537 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs"] Oct 31 19:20:25.097644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:25.097616 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5464e64-b8mcs"] Oct 31 19:20:25.567576 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:25.567538 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0da11b-3490-452c-8c1c-b346274e32c3" path="/var/lib/kubelet/pods/8a0da11b-3490-452c-8c1c-b346274e32c3/volumes" Oct 31 19:20:32.004256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.004225 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc"] Oct 31 19:20:32.009457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.009439 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.112398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.112365 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-150680-serving-cert\"" Oct 31 19:20:32.115056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.115035 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc"] Oct 31 19:20:32.133922 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.133888 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.134054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.133954 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.134156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.134135 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xpv\" (UniqueName: \"kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.134252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.134220 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.235290 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.235253 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.235461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.235347 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xpv\" (UniqueName: \"kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.235461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.235373 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.235461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.235421 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.235635 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:20:32.235546 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-serving-cert: secret "maas-api-e2e-extmod-a-150680-serving-cert" not found Oct 31 19:20:32.235635 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:20:32.235620 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls podName:fae0bf45-bd09-4266-af6d-28d56a07117b nodeName:}" failed. No retries permitted until 2026-10-31 19:20:32.735597894 +0000 UTC m=+3553.337739020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls") pod "maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" (UID: "fae0bf45-bd09-4266-af6d-28d56a07117b") : secret "maas-api-e2e-extmod-a-150680-serving-cert" not found Oct 31 19:20:32.235910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.235891 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.237872 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.237855 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.293685 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.293617 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xpv\" (UniqueName: \"kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.740150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.740117 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.742476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.742456 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:32.923191 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:32.923163 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:33.299991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:33.299969 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc"] Oct 31 19:20:33.302119 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:20:33.302093 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae0bf45_bd09_4266_af6d_28d56a07117b.slice/crio-a69922beef8046f8e74587764d4665befc66506daa5b5659866e944d92bf248c WatchSource:0}: Error finding container a69922beef8046f8e74587764d4665befc66506daa5b5659866e944d92bf248c: Status 404 returned error can't find the container with id a69922beef8046f8e74587764d4665befc66506daa5b5659866e944d92bf248c Oct 31 19:20:33.504924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:33.504894 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" event={"ID":"fae0bf45-bd09-4266-af6d-28d56a07117b","Type":"ContainerStarted","Data":"a69922beef8046f8e74587764d4665befc66506daa5b5659866e944d92bf248c"} Oct 31 19:20:34.511761 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:34.511718 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" event={"ID":"fae0bf45-bd09-4266-af6d-28d56a07117b","Type":"ContainerStarted","Data":"e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11"} Oct 31 19:20:34.512138 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:34.511785 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:34.631280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:34.631235 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" podStartSLOduration=2.821304827 podStartE2EDuration="3.631221425s" totalImagesPullingTime="809.916598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:34.628354109 +0000 UTC m=+3555.230495253" watchObservedRunningTime="2026-10-31 19:20:34.631221425 +0000 UTC m=+3555.233362568" Oct 31 19:20:40.522926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:40.522896 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:20:41.476305 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.476271 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps"] Oct 31 19:20:41.480139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.480120 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.529997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.529964 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530007 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530030 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530153 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530288 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530327 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9rn\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530414 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530445 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.530822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.530475 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.533954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.533935 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9cf479-openshift-default-dockercfg-9j62d\"" Oct 31 19:20:41.631698 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631662 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.631891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631708 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.631891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631745 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.631891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631783 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.631891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631821 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.631891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631846 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.631891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631888 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.631971 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.632018 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9rn\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.632131 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632378 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.632324 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.632566 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632724 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.632650 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.632794 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.632775 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.634151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.634133 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.634490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.634473 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.687793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.687759 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps"] Oct 31 19:20:41.691219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.690442 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.691219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.691046 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9rn\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:41.793354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:41.793325 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:42.227219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.227179 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps"] Oct 31 19:20:42.229614 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:20:42.229583 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e49b6fc_0648_4079_9048_b24ca7dd097c.slice/crio-492093189af2582dfd3e89be515231bea45affba8060a91d4f4d5a7188417a52 WatchSource:0}: Error finding container 492093189af2582dfd3e89be515231bea45affba8060a91d4f4d5a7188417a52: Status 404 returned error can't find the container with id 492093189af2582dfd3e89be515231bea45affba8060a91d4f4d5a7188417a52 Oct 31 19:20:42.232124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.232085 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:20:42.232236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.232154 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:20:42.232236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.232219 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:20:42.562326 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.562277 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" event={"ID":"7e49b6fc-0648-4079-9048-b24ca7dd097c","Type":"ContainerStarted","Data":"19ed436a4cd75ed4417bdb1a6ab325a46bc85fe345a458b80aedd1285db594bf"} Oct 31 19:20:42.562326 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.562332 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" event={"ID":"7e49b6fc-0648-4079-9048-b24ca7dd097c","Type":"ContainerStarted","Data":"492093189af2582dfd3e89be515231bea45affba8060a91d4f4d5a7188417a52"} Oct 31 19:20:42.683473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.683423 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podStartSLOduration=1.683409611 podStartE2EDuration="1.683409611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:42.68110697 +0000 UTC m=+3563.283248114" watchObservedRunningTime="2026-10-31 19:20:42.683409611 +0000 UTC m=+3563.285550754" Oct 31 19:20:42.793664 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:42.793633 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:43.795216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:43.795165 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 19:20:43.795597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:43.795244 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 19:20:44.794029 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:44.794000 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 19:20:44.794301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:44.794056 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 19:20:45.794167 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:45.794127 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 19:20:45.794663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:45.794212 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 19:20:46.794048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:46.794004 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 19:20:46.794269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:46.794079 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 19:20:47.794596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:47.794560 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 19:20:47.794997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:47.794609 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 19:20:48.798637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:48.798609 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:48.799102 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:48.799082 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:48.799875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:20:48.799856 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:20:54.904341 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:20:54.904306 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:21:20.120833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:20.120801 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:21:20.121793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:20.121770 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:21:21.401024 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:21.400997 5688 scope.go:122] "RemoveContainer" containerID="8791c31b22ce4ba8c15c27fa6381287431ddc61163d083588b0f6e5ead2d12c0" Oct 31 19:21:21.413069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:21.413049 5688 scope.go:122] "RemoveContainer" containerID="0caa60c2366915c30e68a1ab58b8e0e64dab9f7e00fe286ea553b558438ee0b5" Oct 31 19:21:21.421686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:21.421668 5688 scope.go:122] "RemoveContainer" containerID="a14d6b1a1d0e5cf31ea1db13ca06cb2a623638b5a7a953016bb067751f0ddbcd" Oct 31 19:21:24.383375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.383342 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz"] Oct 31 19:21:24.388644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.388629 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.483440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.483398 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.483440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.483449 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.483674 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.483562 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.483730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.483680 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvwk\" (UniqueName: \"kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.494729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.494702 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-757472-serving-cert\"" Oct 31 19:21:24.497122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.497104 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz"] Oct 31 19:21:24.584950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.584910 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.584950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.584952 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.585234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.584973 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.585234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.585009 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvwk\" (UniqueName: \"kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.585234 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:21:24.585108 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-757472-serving-cert: secret "maas-api-e2e-autores-757472-serving-cert" not found Oct 31 19:21:24.585234 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:21:24.585225 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls podName:1adb78bf-8ad0-416e-bd49-e50740a5e449 nodeName:}" failed. No retries permitted until 2026-10-31 19:21:25.085173035 +0000 UTC m=+3605.687314173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls") pod "maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" (UID: "1adb78bf-8ad0-416e-bd49-e50740a5e449") : secret "maas-api-e2e-autores-757472-serving-cert" not found Oct 31 19:21:24.585628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.585612 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.587441 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.587425 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.647829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:24.647749 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvwk\" (UniqueName: \"kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:24.905194 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:21:24.905103 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:21:25.089660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:25.089630 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:25.092046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:25.092016 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\") pod \"maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:25.302960 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:25.302929 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:25.676962 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:25.676940 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz"] Oct 31 19:21:25.678827 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:21:25.678791 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adb78bf_8ad0_416e_bd49_e50740a5e449.slice/crio-c0cad78101188ea1c9e36711206d744f9bb1ac39be57871a7b5bfa9def7cf052 WatchSource:0}: Error finding container c0cad78101188ea1c9e36711206d744f9bb1ac39be57871a7b5bfa9def7cf052: Status 404 returned error can't find the container with id c0cad78101188ea1c9e36711206d744f9bb1ac39be57871a7b5bfa9def7cf052 Oct 31 19:21:25.813927 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:25.813897 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" event={"ID":"1adb78bf-8ad0-416e-bd49-e50740a5e449","Type":"ContainerStarted","Data":"c0cad78101188ea1c9e36711206d744f9bb1ac39be57871a7b5bfa9def7cf052"} Oct 31 19:21:26.821609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:26.821535 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" event={"ID":"1adb78bf-8ad0-416e-bd49-e50740a5e449","Type":"ContainerStarted","Data":"d453ab3c709c83f56425ce9ae3c5f7cb57434356cac8edb8b8c4ce2b04f29f89"} Oct 31 19:21:26.821957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:26.821698 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:26.937126 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:26.937075 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" podStartSLOduration=2.076873943 podStartE2EDuration="2.9370613s" totalImagesPullingTime="860.187357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:26.934847117 +0000 UTC m=+3607.536988260" watchObservedRunningTime="2026-10-31 19:21:26.9370613 +0000 UTC m=+3607.539202443" Oct 31 19:21:32.832392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:32.832302 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:21:49.094199 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:49.094156 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs"] Oct 31 19:21:49.203804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:49.203767 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7"] Oct 31 19:21:54.095085 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.095034 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" podUID="c17c2cff-4777-4943-8bf7-38e1d5c07d72" containerName="payload-processing" containerID="cri-o://141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8" gracePeriod=25 Oct 31 19:21:54.205112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.205081 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" podUID="20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" containerName="payload-pre-processing" containerID="cri-o://654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150" gracePeriod=25 Oct 31 19:21:54.368346 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.368314 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:21:54.479460 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.479434 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:21:54.495463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.495435 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n\" (UniqueName: \"kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n\") pod \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " Oct 31 19:21:54.495605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.495483 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6\" (UniqueName: \"kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6\") pod \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " Oct 31 19:21:54.495655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.495639 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume\") pod \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\" (UID: \"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd\") " Oct 31 19:21:54.495695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.495676 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume\") pod \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\" (UID: \"c17c2cff-4777-4943-8bf7-38e1d5c07d72\") " Oct 31 19:21:54.496093 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.496061 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume" pod "20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" (UID: "20c53ca5-64e9-46be-ab0b-c68dcc50a4fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:21:54.496427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.496282 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume" pod "c17c2cff-4777-4943-8bf7-38e1d5c07d72" (UID: "c17c2cff-4777-4943-8bf7-38e1d5c07d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:21:54.497639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.497612 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n" pod "c17c2cff-4777-4943-8bf7-38e1d5c07d72" (UID: "c17c2cff-4777-4943-8bf7-38e1d5c07d72"). InnerVolumeSpecName "kube-api-access-6fs5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:54.497739 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.497698 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6" pod "20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" (UID: "20c53ca5-64e9-46be-ab0b-c68dcc50a4fd"). InnerVolumeSpecName "kube-api-access-xdtt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:54.597091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.597005 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17c2cff-4777-4943-8bf7-38e1d5c07d72-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:21:54.597091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.597029 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fs5n\" (UniqueName: \"kubernetes.io/projected/c17c2cff-4777-4943-8bf7-38e1d5c07d72-kube-api-access-6fs5n\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:21:54.597091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.597043 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdtt6\" (UniqueName: \"kubernetes.io/projected/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-kube-api-access-xdtt6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:21:54.597091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:54.597056 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:21:54.903903 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:21:54.903827 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:21:55.001773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.001738 5688 generic.go:356] "Generic (PLEG): container finished" podID="20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" containerID="654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150" exitCode=0 Oct 31 19:21:55.001983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.001783 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" Oct 31 19:21:55.001983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.001824 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" event={"ID":"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd","Type":"ContainerDied","Data":"654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150"} Oct 31 19:21:55.001983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.001871 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7" event={"ID":"20c53ca5-64e9-46be-ab0b-c68dcc50a4fd","Type":"ContainerDied","Data":"5b5caa18e964e9177de12bcfe1334711a6474ba003e44938000e40092bcdc499"} Oct 31 19:21:55.001983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.001897 5688 scope.go:122] "RemoveContainer" containerID="654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150" Oct 31 19:21:55.003430 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.003405 5688 generic.go:356] "Generic (PLEG): container finished" podID="c17c2cff-4777-4943-8bf7-38e1d5c07d72" containerID="141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8" exitCode=0 Oct 31 19:21:55.003564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.003452 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" Oct 31 19:21:55.003564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.003489 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" event={"ID":"c17c2cff-4777-4943-8bf7-38e1d5c07d72","Type":"ContainerDied","Data":"141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8"} Oct 31 19:21:55.003564 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.003549 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs" event={"ID":"c17c2cff-4777-4943-8bf7-38e1d5c07d72","Type":"ContainerDied","Data":"4e9bf2ad803faee9716efc0ad2de958d1ad5f45965c7c2a65b45c21815bce1a2"} Oct 31 19:21:55.012035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.012019 5688 scope.go:122] "RemoveContainer" containerID="654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150" Oct 31 19:21:55.012429 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:21:55.012410 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150\": container with ID starting with 654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150 not found: ID does not exist" containerID="654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150" Oct 31 19:21:55.012507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.012437 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150"} err="failed to get container status \"654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150\": rpc error: code = NotFound desc = could not find container \"654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150\": container with ID starting with 654bfaca68caa95f898ed41784e7eafab07b250fc37f4885560c80dde0679150 not found: ID does not exist" Oct 31 19:21:55.012507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.012458 5688 scope.go:122] "RemoveContainer" containerID="141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8" Oct 31 19:21:55.021360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.021346 5688 scope.go:122] "RemoveContainer" containerID="141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8" Oct 31 19:21:55.021596 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:21:55.021578 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8\": container with ID starting with 141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8 not found: ID does not exist" containerID="141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8" Oct 31 19:21:55.021645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.021603 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8"} err="failed to get container status \"141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8\": rpc error: code = NotFound desc = could not find container \"141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8\": container with ID starting with 141d356fe68fa78a144d5289cf10a2c8ed8e6fd6e8622e641670419cd7c949c8 not found: ID does not exist" Oct 31 19:21:55.166217 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.166143 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7"] Oct 31 19:21:55.170443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.170420 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9cf479-857d87bd7f-7crm7"] Oct 31 19:21:55.278783 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.278757 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs"] Oct 31 19:21:55.281038 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.281013 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9cf479-79b5698f6c-g9jbs"] Oct 31 19:21:55.566667 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.566640 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" path="/var/lib/kubelet/pods/20c53ca5-64e9-46be-ab0b-c68dcc50a4fd/volumes" Oct 31 19:21:55.567014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:21:55.566998 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17c2cff-4777-4943-8bf7-38e1d5c07d72" path="/var/lib/kubelet/pods/c17c2cff-4777-4943-8bf7-38e1d5c07d72/volumes" Oct 31 19:22:04.057364 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:04.057332 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq"] Oct 31 19:22:04.168172 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:04.168134 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5"] Oct 31 19:22:09.058327 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.058288 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" podUID="2a640f64-8ecf-412c-ac97-aa245d9396fb" containerName="payload-processing" containerID="cri-o://6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b" gracePeriod=25 Oct 31 19:22:09.169538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.169498 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" podUID="c54e248d-d756-4fbb-82da-e4db0047b6bf" containerName="payload-pre-processing" containerID="cri-o://7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc" gracePeriod=25 Oct 31 19:22:09.322167 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.322102 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:22:09.440627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.440598 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:22:09.456436 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.456411 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27\" (UniqueName: \"kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27\") pod \"2a640f64-8ecf-412c-ac97-aa245d9396fb\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " Oct 31 19:22:09.456588 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.456504 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume\") pod \"2a640f64-8ecf-412c-ac97-aa245d9396fb\" (UID: \"2a640f64-8ecf-412c-ac97-aa245d9396fb\") " Oct 31 19:22:09.456931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.456905 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume" pod "2a640f64-8ecf-412c-ac97-aa245d9396fb" (UID: "2a640f64-8ecf-412c-ac97-aa245d9396fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:22:09.458717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.458691 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27" pod "2a640f64-8ecf-412c-ac97-aa245d9396fb" (UID: "2a640f64-8ecf-412c-ac97-aa245d9396fb"). InnerVolumeSpecName "kube-api-access-khw27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:22:09.557068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.557037 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume\") pod \"c54e248d-d756-4fbb-82da-e4db0047b6bf\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " Oct 31 19:22:09.557283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.557263 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb\" (UniqueName: \"kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb\") pod \"c54e248d-d756-4fbb-82da-e4db0047b6bf\" (UID: \"c54e248d-d756-4fbb-82da-e4db0047b6bf\") " Oct 31 19:22:09.557411 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.557378 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume" pod "c54e248d-d756-4fbb-82da-e4db0047b6bf" (UID: "c54e248d-d756-4fbb-82da-e4db0047b6bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:22:09.557642 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.557626 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khw27\" (UniqueName: \"kubernetes.io/projected/2a640f64-8ecf-412c-ac97-aa245d9396fb-kube-api-access-khw27\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:09.557717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.557648 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54e248d-d756-4fbb-82da-e4db0047b6bf-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:09.557717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.557659 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a640f64-8ecf-412c-ac97-aa245d9396fb-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:09.559568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.559539 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb" pod "c54e248d-d756-4fbb-82da-e4db0047b6bf" (UID: "c54e248d-d756-4fbb-82da-e4db0047b6bf"). InnerVolumeSpecName "kube-api-access-rcwnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:22:09.599157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.599127 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x"] Oct 31 19:22:09.659062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.659032 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcwnb\" (UniqueName: \"kubernetes.io/projected/c54e248d-d756-4fbb-82da-e4db0047b6bf-kube-api-access-rcwnb\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:09.703613 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:09.703587 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w"] Oct 31 19:22:10.094292 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.094261 5688 generic.go:356] "Generic (PLEG): container finished" podID="2a640f64-8ecf-412c-ac97-aa245d9396fb" containerID="6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b" exitCode=0 Oct 31 19:22:10.094671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.094321 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" Oct 31 19:22:10.094671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.094342 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" event={"ID":"2a640f64-8ecf-412c-ac97-aa245d9396fb","Type":"ContainerDied","Data":"6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b"} Oct 31 19:22:10.094671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.094420 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq" event={"ID":"2a640f64-8ecf-412c-ac97-aa245d9396fb","Type":"ContainerDied","Data":"e08a89ea7877e9b6dec5346b6598f498688998d0aa1b24558c9ef9cf255db34e"} Oct 31 19:22:10.094671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.094446 5688 scope.go:122] "RemoveContainer" containerID="6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b" Oct 31 19:22:10.095789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.095717 5688 generic.go:356] "Generic (PLEG): container finished" podID="c54e248d-d756-4fbb-82da-e4db0047b6bf" containerID="7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc" exitCode=0 Oct 31 19:22:10.095789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.095746 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" event={"ID":"c54e248d-d756-4fbb-82da-e4db0047b6bf","Type":"ContainerDied","Data":"7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc"} Oct 31 19:22:10.095789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.095779 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" Oct 31 19:22:10.095965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.095790 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5" event={"ID":"c54e248d-d756-4fbb-82da-e4db0047b6bf","Type":"ContainerDied","Data":"698b83eaa624c8b6297564c973b60c56ab1e36528edd5516bf00fd64df088c54"} Oct 31 19:22:10.107734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.107668 5688 scope.go:122] "RemoveContainer" containerID="6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b" Oct 31 19:22:10.108212 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:10.108146 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b\": container with ID starting with 6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b not found: ID does not exist" containerID="6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b" Oct 31 19:22:10.108319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.108217 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b"} err="failed to get container status \"6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b\": rpc error: code = NotFound desc = could not find container \"6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b\": container with ID starting with 6e3a14047c3e07d8307b8f0076c19c90d5878cc67831aa5499e46879950d680b not found: ID does not exist" Oct 31 19:22:10.108319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.108235 5688 scope.go:122] "RemoveContainer" containerID="7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc" Oct 31 19:22:10.116701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.116685 5688 scope.go:122] "RemoveContainer" containerID="7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc" Oct 31 19:22:10.116956 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:10.116930 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc\": container with ID starting with 7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc not found: ID does not exist" containerID="7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc" Oct 31 19:22:10.117049 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.116961 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc"} err="failed to get container status \"7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc\": rpc error: code = NotFound desc = could not find container \"7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc\": container with ID starting with 7477f9e55c9765c6344ce532ef54d547dbbb1d5036e32975920c300b9f4caddc not found: ID does not exist" Oct 31 19:22:10.258746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.258720 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5"] Oct 31 19:22:10.262844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.262815 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ced6b-7c6c487d67-qn8m5"] Oct 31 19:22:10.370005 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.369942 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq"] Oct 31 19:22:10.375581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:10.375559 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ced6b-6bf9fbf-xfzqq"] Oct 31 19:22:11.050849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.045141 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc"] Oct 31 19:22:11.050849 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.045421 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" podUID="fae0bf45-bd09-4266-af6d-28d56a07117b" containerName="maas-api" containerID="cri-o://e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11" gracePeriod=30 Oct 31 19:22:11.367695 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.367402 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:22:11.487293 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.486993 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv\" (UniqueName: \"kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv\") pod \"fae0bf45-bd09-4266-af6d-28d56a07117b\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " Oct 31 19:22:11.487293 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.487037 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs\") pod \"fae0bf45-bd09-4266-af6d-28d56a07117b\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " Oct 31 19:22:11.487293 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.487091 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\") pod \"fae0bf45-bd09-4266-af6d-28d56a07117b\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " Oct 31 19:22:11.487293 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.487176 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca\") pod \"fae0bf45-bd09-4266-af6d-28d56a07117b\" (UID: \"fae0bf45-bd09-4266-af6d-28d56a07117b\") " Oct 31 19:22:11.487646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.487541 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca" pod "fae0bf45-bd09-4266-af6d-28d56a07117b" (UID: "fae0bf45-bd09-4266-af6d-28d56a07117b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:22:11.487797 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.487724 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae0bf45-bd09-4266-af6d-28d56a07117b-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:11.489983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.489950 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls" pod "fae0bf45-bd09-4266-af6d-28d56a07117b" (UID: "fae0bf45-bd09-4266-af6d-28d56a07117b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:22:11.490271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.490246 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv" pod "fae0bf45-bd09-4266-af6d-28d56a07117b" (UID: "fae0bf45-bd09-4266-af6d-28d56a07117b"). InnerVolumeSpecName "kube-api-access-g9xpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:22:11.490366 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.490300 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs" pod "fae0bf45-bd09-4266-af6d-28d56a07117b" (UID: "fae0bf45-bd09-4266-af6d-28d56a07117b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:22:11.567848 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.567819 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a640f64-8ecf-412c-ac97-aa245d9396fb" path="/var/lib/kubelet/pods/2a640f64-8ecf-412c-ac97-aa245d9396fb/volumes" Oct 31 19:22:11.568256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.568242 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54e248d-d756-4fbb-82da-e4db0047b6bf" path="/var/lib/kubelet/pods/c54e248d-d756-4fbb-82da-e4db0047b6bf/volumes" Oct 31 19:22:11.589273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.589215 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xpv\" (UniqueName: \"kubernetes.io/projected/fae0bf45-bd09-4266-af6d-28d56a07117b-kube-api-access-g9xpv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:11.589273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.589237 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:11.589273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:11.589246 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae0bf45-bd09-4266-af6d-28d56a07117b-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:12.122837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.122803 5688 generic.go:356] "Generic (PLEG): container finished" podID="fae0bf45-bd09-4266-af6d-28d56a07117b" containerID="e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11" exitCode=0 Oct 31 19:22:12.122989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.122853 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" Oct 31 19:22:12.122989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.122888 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" event={"ID":"fae0bf45-bd09-4266-af6d-28d56a07117b","Type":"ContainerDied","Data":"e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11"} Oct 31 19:22:12.122989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.122947 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc" event={"ID":"fae0bf45-bd09-4266-af6d-28d56a07117b","Type":"ContainerDied","Data":"a69922beef8046f8e74587764d4665befc66506daa5b5659866e944d92bf248c"} Oct 31 19:22:12.122989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.122969 5688 scope.go:122] "RemoveContainer" containerID="e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11" Oct 31 19:22:12.132753 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.132703 5688 scope.go:122] "RemoveContainer" containerID="e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11" Oct 31 19:22:12.133020 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:12.133000 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11\": container with ID starting with e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11 not found: ID does not exist" containerID="e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11" Oct 31 19:22:12.133091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.133027 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11"} err="failed to get container status \"e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11\": rpc error: code = NotFound desc = could not find container \"e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11\": container with ID starting with e7ed90d6de10dac7a30621e54a4ab9f921d64324856f77167e8aaa0748635f11 not found: ID does not exist" Oct 31 19:22:12.288244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.288212 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc"] Oct 31 19:22:12.337904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:12.337872 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-150680-698f9c4fdc-cgccc"] Oct 31 19:22:13.566233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:13.566202 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae0bf45-bd09-4266-af6d-28d56a07117b" path="/var/lib/kubelet/pods/fae0bf45-bd09-4266-af6d-28d56a07117b/volumes" Oct 31 19:22:14.600235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:14.600178 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" podUID="457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" containerName="payload-processing" containerID="cri-o://87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3" gracePeriod=25 Oct 31 19:22:14.704660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:14.704626 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" podUID="154938fe-a662-4607-9f25-197e025f7953" containerName="payload-pre-processing" containerID="cri-o://72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9" gracePeriod=25 Oct 31 19:22:14.866112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:14.866087 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:22:14.960936 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:14.960915 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:22:15.020397 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.020369 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume\") pod \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " Oct 31 19:22:15.020549 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.020510 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks\" (UniqueName: \"kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks\") pod \"154938fe-a662-4607-9f25-197e025f7953\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " Oct 31 19:22:15.020596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.020578 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume\") pod \"154938fe-a662-4607-9f25-197e025f7953\" (UID: \"154938fe-a662-4607-9f25-197e025f7953\") " Oct 31 19:22:15.020632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.020623 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn\" (UniqueName: \"kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn\") pod \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\" (UID: \"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b\") " Oct 31 19:22:15.020726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.020704 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume" pod "457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" (UID: "457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:22:15.020978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.020956 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume" pod "154938fe-a662-4607-9f25-197e025f7953" (UID: "154938fe-a662-4607-9f25-197e025f7953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:22:15.021204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.021161 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:15.021204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.021200 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154938fe-a662-4607-9f25-197e025f7953-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:15.022835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.022813 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn" pod "457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" (UID: "457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b"). InnerVolumeSpecName "kube-api-access-7qmxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:22:15.022888 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.022827 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks" pod "154938fe-a662-4607-9f25-197e025f7953" (UID: "154938fe-a662-4607-9f25-197e025f7953"). InnerVolumeSpecName "kube-api-access-626ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:22:15.121966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.121902 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626ks\" (UniqueName: \"kubernetes.io/projected/154938fe-a662-4607-9f25-197e025f7953-kube-api-access-626ks\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:15.121966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.121931 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qmxn\" (UniqueName: \"kubernetes.io/projected/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b-kube-api-access-7qmxn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:22:15.142691 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.142662 5688 generic.go:356] "Generic (PLEG): container finished" podID="457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" containerID="87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3" exitCode=0 Oct 31 19:22:15.142831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.142712 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" Oct 31 19:22:15.142831 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.142750 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" event={"ID":"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b","Type":"ContainerDied","Data":"87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3"} Oct 31 19:22:15.142952 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.142833 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x" event={"ID":"457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b","Type":"ContainerDied","Data":"75aecd8d4dc3837e53fc29b38db35eeca084b6ff8d80f28f46c58c9980b0cce7"} Oct 31 19:22:15.142952 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.142863 5688 scope.go:122] "RemoveContainer" containerID="87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3" Oct 31 19:22:15.144288 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.144265 5688 generic.go:356] "Generic (PLEG): container finished" podID="154938fe-a662-4607-9f25-197e025f7953" containerID="72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9" exitCode=0 Oct 31 19:22:15.144392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.144307 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" Oct 31 19:22:15.144392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.144316 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" event={"ID":"154938fe-a662-4607-9f25-197e025f7953","Type":"ContainerDied","Data":"72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9"} Oct 31 19:22:15.144392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.144357 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w" event={"ID":"154938fe-a662-4607-9f25-197e025f7953","Type":"ContainerDied","Data":"49151b4dd979f123e4340f1287f49097ef0db7c93bdc6c14e52db4e01b78ce46"} Oct 31 19:22:15.154316 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.154295 5688 scope.go:122] "RemoveContainer" containerID="87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3" Oct 31 19:22:15.154587 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:15.154568 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3\": container with ID starting with 87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3 not found: ID does not exist" containerID="87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3" Oct 31 19:22:15.154647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.154596 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3"} err="failed to get container status \"87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3\": rpc error: code = NotFound desc = could not find container \"87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3\": container with ID starting with 87ec60a4f475b99f93e799a47c8fafb25ad3311d49e2436cb6274aa3e219ecf3 not found: ID does not exist" Oct 31 19:22:15.154647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.154613 5688 scope.go:122] "RemoveContainer" containerID="72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9" Oct 31 19:22:15.164039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.164023 5688 scope.go:122] "RemoveContainer" containerID="72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9" Oct 31 19:22:15.164325 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:15.164308 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9\": container with ID starting with 72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9 not found: ID does not exist" containerID="72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9" Oct 31 19:22:15.164429 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.164331 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9"} err="failed to get container status \"72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9\": rpc error: code = NotFound desc = could not find container \"72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9\": container with ID starting with 72e6add78e8c44728d867eb4b0a1c8b89f143c2c0ad13821241913ca7771aff9 not found: ID does not exist" Oct 31 19:22:15.306701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.306674 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w"] Oct 31 19:22:15.310050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.310026 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-150680-56f5bb97dc-8t62w"] Oct 31 19:22:15.418210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.418139 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x"] Oct 31 19:22:15.421452 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.421424 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-150680-766c795bdc-28s6x"] Oct 31 19:22:15.566135 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.566110 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154938fe-a662-4607-9f25-197e025f7953" path="/var/lib/kubelet/pods/154938fe-a662-4607-9f25-197e025f7953/volumes" Oct 31 19:22:15.566502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:15.566489 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" path="/var/lib/kubelet/pods/457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b/volumes" Oct 31 19:22:21.451168 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:21.451136 5688 scope.go:122] "RemoveContainer" containerID="7f3d288836ece63c7bfbe3bbbfd1d4ca88a0c6abface0e372cf00a4ff668cde1" Oct 31 19:22:24.904819 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:24.904775 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:22:30.047679 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:30.047622 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp"] Oct 31 19:22:30.081194 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:30.081144 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a163ff-v86pp"] Oct 31 19:22:31.566154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:31.566123 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9a4fbd-efbe-43d0-aefe-ed798d04d3df" path="/var/lib/kubelet/pods/ce9a4fbd-efbe-43d0-aefe-ed798d04d3df/volumes" Oct 31 19:22:41.047230 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:41.047177 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz"] Oct 31 19:22:41.096122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:41.096094 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-414a52fd-z2bkz"] Oct 31 19:22:41.566542 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:22:41.566509 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd674ae-b74b-45fb-a95f-fd5333092a28" path="/var/lib/kubelet/pods/dcd674ae-b74b-45fb-a95f-fd5333092a28/volumes" Oct 31 19:22:54.906468 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:22:54.906340 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:23:00.265416 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.265342 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz"] Oct 31 19:23:00.265836 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.265547 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" podUID="1adb78bf-8ad0-416e-bd49-e50740a5e449" containerName="maas-api" containerID="cri-o://d453ab3c709c83f56425ce9ae3c5f7cb57434356cac8edb8b8c4ce2b04f29f89" gracePeriod=30 Oct 31 19:23:00.431985 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.431948 5688 generic.go:356] "Generic (PLEG): container finished" podID="1adb78bf-8ad0-416e-bd49-e50740a5e449" containerID="d453ab3c709c83f56425ce9ae3c5f7cb57434356cac8edb8b8c4ce2b04f29f89" exitCode=0 Oct 31 19:23:00.432155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.432020 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" event={"ID":"1adb78bf-8ad0-416e-bd49-e50740a5e449","Type":"ContainerDied","Data":"d453ab3c709c83f56425ce9ae3c5f7cb57434356cac8edb8b8c4ce2b04f29f89"} Oct 31 19:23:00.602330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.602301 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:23:00.640035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.640002 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs\") pod \"1adb78bf-8ad0-416e-bd49-e50740a5e449\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " Oct 31 19:23:00.640208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.640088 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\") pod \"1adb78bf-8ad0-416e-bd49-e50740a5e449\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " Oct 31 19:23:00.640267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.640204 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca\") pod \"1adb78bf-8ad0-416e-bd49-e50740a5e449\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " Oct 31 19:23:00.640364 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.640344 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk\" (UniqueName: \"kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk\") pod \"1adb78bf-8ad0-416e-bd49-e50740a5e449\" (UID: \"1adb78bf-8ad0-416e-bd49-e50740a5e449\") " Oct 31 19:23:00.640574 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.640537 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca" pod "1adb78bf-8ad0-416e-bd49-e50740a5e449" (UID: "1adb78bf-8ad0-416e-bd49-e50740a5e449"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:23:00.640877 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.640857 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb78bf-8ad0-416e-bd49-e50740a5e449-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:00.643120 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.643083 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs" pod "1adb78bf-8ad0-416e-bd49-e50740a5e449" (UID: "1adb78bf-8ad0-416e-bd49-e50740a5e449"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:23:00.643378 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.643330 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls" pod "1adb78bf-8ad0-416e-bd49-e50740a5e449" (UID: "1adb78bf-8ad0-416e-bd49-e50740a5e449"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:23:00.643973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.643460 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk" pod "1adb78bf-8ad0-416e-bd49-e50740a5e449" (UID: "1adb78bf-8ad0-416e-bd49-e50740a5e449"). InnerVolumeSpecName "kube-api-access-kwvwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:00.741878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.741839 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:00.741878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.741875 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwvwk\" (UniqueName: \"kubernetes.io/projected/1adb78bf-8ad0-416e-bd49-e50740a5e449-kube-api-access-kwvwk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:00.742127 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:00.741894 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb78bf-8ad0-416e-bd49-e50740a5e449-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:01.438402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:01.438377 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" Oct 31 19:23:01.438926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:01.438405 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz" event={"ID":"1adb78bf-8ad0-416e-bd49-e50740a5e449","Type":"ContainerDied","Data":"c0cad78101188ea1c9e36711206d744f9bb1ac39be57871a7b5bfa9def7cf052"} Oct 31 19:23:01.438926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:01.438458 5688 scope.go:122] "RemoveContainer" containerID="d453ab3c709c83f56425ce9ae3c5f7cb57434356cac8edb8b8c4ce2b04f29f89" Oct 31 19:23:01.610326 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:01.610300 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz"] Oct 31 19:23:01.659818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:01.659789 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-757472-6bdcdbbfb9-tm6fz"] Oct 31 19:23:03.566171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:03.566133 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adb78bf-8ad0-416e-bd49-e50740a5e449" path="/var/lib/kubelet/pods/1adb78bf-8ad0-416e-bd49-e50740a5e449/volumes" Oct 31 19:23:09.017702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.017665 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8"] Oct 31 19:23:09.018155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018142 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c54e248d-d756-4fbb-82da-e4db0047b6bf" containerName="payload-pre-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018155 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54e248d-d756-4fbb-82da-e4db0047b6bf" containerName="payload-pre-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018166 5688 cpu_manager.go:414] "containerMap: removing container" podUID="457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" containerName="payload-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018171 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" containerName="payload-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018196 5688 cpu_manager.go:414] "containerMap: removing container" podUID="154938fe-a662-4607-9f25-197e025f7953" containerName="payload-pre-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018202 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154938fe-a662-4607-9f25-197e025f7953" containerName="payload-pre-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018223 5688 cpu_manager.go:414] "containerMap: removing container" podUID="20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" containerName="payload-pre-processing" Oct 31 19:23:09.018227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018227 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" containerName="payload-pre-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018239 5688 cpu_manager.go:414] "containerMap: removing container" podUID="fae0bf45-bd09-4266-af6d-28d56a07117b" containerName="maas-api" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018244 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae0bf45-bd09-4266-af6d-28d56a07117b" containerName="maas-api" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018250 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c17c2cff-4777-4943-8bf7-38e1d5c07d72" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018254 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17c2cff-4777-4943-8bf7-38e1d5c07d72" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018266 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1adb78bf-8ad0-416e-bd49-e50740a5e449" containerName="maas-api" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018270 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adb78bf-8ad0-416e-bd49-e50740a5e449" containerName="maas-api" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018278 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2a640f64-8ecf-412c-ac97-aa245d9396fb" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018283 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a640f64-8ecf-412c-ac97-aa245d9396fb" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018353 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17c2cff-4777-4943-8bf7-38e1d5c07d72" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018363 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adb78bf-8ad0-416e-bd49-e50740a5e449" containerName="maas-api" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018370 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a640f64-8ecf-412c-ac97-aa245d9396fb" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018380 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54e248d-d756-4fbb-82da-e4db0047b6bf" containerName="payload-pre-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018385 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="457a6bd6-dfb3-4e0e-b81f-5ba6c295fe0b" containerName="payload-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018390 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="154938fe-a662-4607-9f25-197e025f7953" containerName="payload-pre-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018398 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c53ca5-64e9-46be-ab0b-c68dcc50a4fd" containerName="payload-pre-processing" Oct 31 19:23:09.018447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.018404 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae0bf45-bd09-4266-af6d-28d56a07117b" containerName="maas-api" Oct 31 19:23:09.023027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.023008 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.078246 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.075590 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b70248-openshift-default-dockercfg-rsb54\"" Oct 31 19:23:09.123254 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123216 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123357 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123534 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123558 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123588 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123604 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rwf\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123624 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123668 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.123845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.123704 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.149727 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.149686 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8"] Oct 31 19:23:09.225425 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225393 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225447 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225479 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225513 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225535 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rwf\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225562 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225620 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225953 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225677 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225953 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225753 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.225953 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.225879 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.226129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.226090 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.232259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.226542 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.232259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.226656 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.232259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.226787 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.232259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.228713 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.232259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.230603 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.297811 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.297733 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.297964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.297941 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rwf\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf\") pod \"e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.337125 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.337093 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:09.800554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.800519 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8"] Oct 31 19:23:09.802491 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:23:09.802455 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f134f9d_1862_48db_95d4_843d1a967dd9.slice/crio-8e380952d1cb7ca33d82ae84bdc375c43953cac06ff45cf55320319be818c800 WatchSource:0}: Error finding container 8e380952d1cb7ca33d82ae84bdc375c43953cac06ff45cf55320319be818c800: Status 404 returned error can't find the container with id 8e380952d1cb7ca33d82ae84bdc375c43953cac06ff45cf55320319be818c800 Oct 31 19:23:09.805632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.805595 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:23:09.805748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.805662 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:23:09.805748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:09.805689 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:23:10.492641 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:10.492599 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" event={"ID":"7f134f9d-1862-48db-95d4-843d1a967dd9","Type":"ContainerStarted","Data":"c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220"} Oct 31 19:23:10.493117 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:10.492654 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" event={"ID":"7f134f9d-1862-48db-95d4-843d1a967dd9","Type":"ContainerStarted","Data":"8e380952d1cb7ca33d82ae84bdc375c43953cac06ff45cf55320319be818c800"} Oct 31 19:23:10.665042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:10.664887 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" podStartSLOduration=2.664872212 podStartE2EDuration="2.664872212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:10.663358064 +0000 UTC m=+3711.265499208" watchObservedRunningTime="2026-10-31 19:23:10.664872212 +0000 UTC m=+3711.267013359" Oct 31 19:23:11.338059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:11.338028 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:11.343838 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:11.343812 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:11.501273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:11.501239 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:11.502705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:11.502681 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:23:14.742785 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.742749 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5"] Oct 31 19:23:14.746607 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.746592 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.783147 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.783118 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhn62\" (UniqueName: \"kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.783305 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.783259 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.783367 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.783339 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.808390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.808365 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5"] Oct 31 19:23:14.849392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.849367 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:23:14.849517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.849372 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:23:14.884511 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.884480 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.884643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.884581 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.884643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.884631 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhn62\" (UniqueName: \"kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.885153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.885129 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.886957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.886941 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:14.943121 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:14.943089 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhn62\" (UniqueName: \"kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62\") pod \"maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:15.060094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:15.060024 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:15.443966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:15.443896 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5"] Oct 31 19:23:15.447414 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:23:15.447387 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda5ebe5_0759_485d_b224_92244204dd16.slice/crio-d2d776abd1cc772a08ac8d4ce201e550568f37423ee1cb7cd4074d7686ea56d4 WatchSource:0}: Error finding container d2d776abd1cc772a08ac8d4ce201e550568f37423ee1cb7cd4074d7686ea56d4: Status 404 returned error can't find the container with id d2d776abd1cc772a08ac8d4ce201e550568f37423ee1cb7cd4074d7686ea56d4 Oct 31 19:23:15.522690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:15.522651 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" event={"ID":"dda5ebe5-0759-485d-b224-92244204dd16","Type":"ContainerStarted","Data":"d2d776abd1cc772a08ac8d4ce201e550568f37423ee1cb7cd4074d7686ea56d4"} Oct 31 19:23:16.529957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:16.529920 5688 generic.go:356] "Generic (PLEG): container finished" podID="dda5ebe5-0759-485d-b224-92244204dd16" containerID="5babd879680216392f0d1bfbffecd098fd4f771bb4b2d7e854bbe9712a605a76" exitCode=0 Oct 31 19:23:16.530438 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:16.530005 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" event={"ID":"dda5ebe5-0759-485d-b224-92244204dd16","Type":"ContainerDied","Data":"5babd879680216392f0d1bfbffecd098fd4f771bb4b2d7e854bbe9712a605a76"} Oct 31 19:23:17.345586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.345551 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t"] Oct 31 19:23:17.349479 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.349459 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.407251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.407221 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b70248-dockercfg-pdbfc\"" Oct 31 19:23:17.407251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.407233 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b70248\"" Oct 31 19:23:17.409660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.409639 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.409763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.409747 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828wt\" (UniqueName: \"kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt\") pod \"payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.455482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.455444 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t"] Oct 31 19:23:17.484341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.484308 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4"] Oct 31 19:23:17.488851 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.488832 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.510665 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.510642 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828wt\" (UniqueName: \"kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt\") pod \"payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.510784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.510751 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.511329 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.511312 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.551571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.551541 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4"] Oct 31 19:23:17.604275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.598904 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828wt\" (UniqueName: \"kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt\") pod \"payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.617137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.617100 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.617526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.617503 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8xz\" (UniqueName: \"kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz\") pod \"payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.665557 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.665524 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:17.722177 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.720514 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8xz\" (UniqueName: \"kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz\") pod \"payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.722177 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.720626 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.722177 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.721335 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.804958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.790892 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8xz\" (UniqueName: \"kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz\") pod \"payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.804958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.804633 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:17.874194 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.874149 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:17.923912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.923407 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62\" (UniqueName: \"kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62\") pod \"dda5ebe5-0759-485d-b224-92244204dd16\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " Oct 31 19:23:17.923912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.923485 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca\") pod \"dda5ebe5-0759-485d-b224-92244204dd16\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " Oct 31 19:23:17.923912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.923507 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca\") pod \"dda5ebe5-0759-485d-b224-92244204dd16\" (UID: \"dda5ebe5-0759-485d-b224-92244204dd16\") " Oct 31 19:23:17.924460 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.924421 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca" pod "dda5ebe5-0759-485d-b224-92244204dd16" (UID: "dda5ebe5-0759-485d-b224-92244204dd16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:23:17.930987 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.929734 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62" pod "dda5ebe5-0759-485d-b224-92244204dd16" (UID: "dda5ebe5-0759-485d-b224-92244204dd16"). InnerVolumeSpecName "kube-api-access-vhn62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:17.930987 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:17.929816 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca" pod "dda5ebe5-0759-485d-b224-92244204dd16" (UID: "dda5ebe5-0759-485d-b224-92244204dd16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:23:18.025464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.025429 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhn62\" (UniqueName: \"kubernetes.io/projected/dda5ebe5-0759-485d-b224-92244204dd16-kube-api-access-vhn62\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:18.025464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.025465 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:18.025652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.025482 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda5ebe5-0759-485d-b224-92244204dd16-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:18.126282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.126204 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t"] Oct 31 19:23:18.131158 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:23:18.131135 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9765f9b2_5824_425e_b2bb_7fb96955a640.slice/crio-e60bac708c51acaf33dbd9f87633d5402bb680d1041d02e4774ec8ed6b6a5a90 WatchSource:0}: Error finding container e60bac708c51acaf33dbd9f87633d5402bb680d1041d02e4774ec8ed6b6a5a90: Status 404 returned error can't find the container with id e60bac708c51acaf33dbd9f87633d5402bb680d1041d02e4774ec8ed6b6a5a90 Oct 31 19:23:18.252605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.252581 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4"] Oct 31 19:23:18.252845 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:23:18.252824 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676ae71a_496c_4164_9aeb_59ff3590afc3.slice/crio-d024455549d6cfde1c29cbaed88c509cc854515eb03d817f3392f61245dd7bc4 WatchSource:0}: Error finding container d024455549d6cfde1c29cbaed88c509cc854515eb03d817f3392f61245dd7bc4: Status 404 returned error can't find the container with id d024455549d6cfde1c29cbaed88c509cc854515eb03d817f3392f61245dd7bc4 Oct 31 19:23:18.544504 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.544471 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" event={"ID":"9765f9b2-5824-425e-b2bb-7fb96955a640","Type":"ContainerStarted","Data":"a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6"} Oct 31 19:23:18.544723 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.544527 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" event={"ID":"9765f9b2-5824-425e-b2bb-7fb96955a640","Type":"ContainerStarted","Data":"e60bac708c51acaf33dbd9f87633d5402bb680d1041d02e4774ec8ed6b6a5a90"} Oct 31 19:23:18.544723 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.544640 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:18.549701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.549667 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" event={"ID":"676ae71a-496c-4164-9aeb-59ff3590afc3","Type":"ContainerStarted","Data":"a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab"} Oct 31 19:23:18.549881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.549731 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:18.549881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.549780 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" event={"ID":"676ae71a-496c-4164-9aeb-59ff3590afc3","Type":"ContainerStarted","Data":"d024455549d6cfde1c29cbaed88c509cc854515eb03d817f3392f61245dd7bc4"} Oct 31 19:23:18.551271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.551251 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" Oct 31 19:23:18.551400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.551277 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5" event={"ID":"dda5ebe5-0759-485d-b224-92244204dd16","Type":"ContainerDied","Data":"d2d776abd1cc772a08ac8d4ce201e550568f37423ee1cb7cd4074d7686ea56d4"} Oct 31 19:23:18.551400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.551307 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d776abd1cc772a08ac8d4ce201e550568f37423ee1cb7cd4074d7686ea56d4" Oct 31 19:23:18.666737 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.666690 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" podStartSLOduration=1.6666754099999999 podStartE2EDuration="1.66667541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:18.66467177 +0000 UTC m=+3719.266812914" watchObservedRunningTime="2026-10-31 19:23:18.66667541 +0000 UTC m=+3719.268816586" Oct 31 19:23:18.802250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:18.802111 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" podStartSLOduration=1.8020913969999999 podStartE2EDuration="1.802091397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:18.801242145 +0000 UTC m=+3719.403383290" watchObservedRunningTime="2026-10-31 19:23:18.802091397 +0000 UTC m=+3719.404232552" Oct 31 19:23:24.905276 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:23:24.905241 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:23:29.569504 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:29.569475 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:23:29.569907 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:29.569542 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:23:30.698541 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:30.698505 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg"] Oct 31 19:23:30.825645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:30.825617 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp"] Oct 31 19:23:32.048778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:32.048744 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx"] Oct 31 19:23:32.051164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:32.051142 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb7b641-grbrx"] Oct 31 19:23:33.566161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:33.566119 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf35085c-eb02-4963-8f12-b50474554c9a" path="/var/lib/kubelet/pods/bf35085c-eb02-4963-8f12-b50474554c9a/volumes" Oct 31 19:23:35.699760 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:35.699717 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" podUID="5276e29e-70b6-42e6-bdf2-f4f859ceaf08" containerName="payload-processing" containerID="cri-o://908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b" gracePeriod=25 Oct 31 19:23:35.826490 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:35.826442 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" podUID="420222b8-86cc-4534-8323-7a6092b6e09f" containerName="payload-pre-processing" containerID="cri-o://351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b" gracePeriod=25 Oct 31 19:23:35.967131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:35.967071 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:23:36.119611 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.119586 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm\" (UniqueName: \"kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm\") pod \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " Oct 31 19:23:36.119757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.119645 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume\") pod \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\" (UID: \"5276e29e-70b6-42e6-bdf2-f4f859ceaf08\") " Oct 31 19:23:36.120006 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.119982 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume" pod "5276e29e-70b6-42e6-bdf2-f4f859ceaf08" (UID: "5276e29e-70b6-42e6-bdf2-f4f859ceaf08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:23:36.121109 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.121091 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:23:36.121798 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.121780 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm" pod "5276e29e-70b6-42e6-bdf2-f4f859ceaf08" (UID: "5276e29e-70b6-42e6-bdf2-f4f859ceaf08"). InnerVolumeSpecName "kube-api-access-b2scm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:36.220558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.220510 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume\") pod \"420222b8-86cc-4534-8323-7a6092b6e09f\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " Oct 31 19:23:36.220558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.220540 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk\" (UniqueName: \"kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk\") pod \"420222b8-86cc-4534-8323-7a6092b6e09f\" (UID: \"420222b8-86cc-4534-8323-7a6092b6e09f\") " Oct 31 19:23:36.220774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.220761 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2scm\" (UniqueName: \"kubernetes.io/projected/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-kube-api-access-b2scm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:36.220774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.220775 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5276e29e-70b6-42e6-bdf2-f4f859ceaf08-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:36.220911 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.220811 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume" pod "420222b8-86cc-4534-8323-7a6092b6e09f" (UID: "420222b8-86cc-4534-8323-7a6092b6e09f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:23:36.222625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.222605 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk" pod "420222b8-86cc-4534-8323-7a6092b6e09f" (UID: "420222b8-86cc-4534-8323-7a6092b6e09f"). InnerVolumeSpecName "kube-api-access-lhwvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:36.322273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.322250 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420222b8-86cc-4534-8323-7a6092b6e09f-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:36.322273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.322271 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhwvk\" (UniqueName: \"kubernetes.io/projected/420222b8-86cc-4534-8323-7a6092b6e09f-kube-api-access-lhwvk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:23:36.660267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.660229 5688 generic.go:356] "Generic (PLEG): container finished" podID="420222b8-86cc-4534-8323-7a6092b6e09f" containerID="351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b" exitCode=0 Oct 31 19:23:36.660267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.660270 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" Oct 31 19:23:36.660519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.660308 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" event={"ID":"420222b8-86cc-4534-8323-7a6092b6e09f","Type":"ContainerDied","Data":"351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b"} Oct 31 19:23:36.660519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.660388 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp" event={"ID":"420222b8-86cc-4534-8323-7a6092b6e09f","Type":"ContainerDied","Data":"3942ff8512b27cffad3f82b3d8338438c7f5f0e08f824fb77924aac1f672869d"} Oct 31 19:23:36.660519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.660420 5688 scope.go:122] "RemoveContainer" containerID="351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b" Oct 31 19:23:36.661597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.661577 5688 generic.go:356] "Generic (PLEG): container finished" podID="5276e29e-70b6-42e6-bdf2-f4f859ceaf08" containerID="908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b" exitCode=0 Oct 31 19:23:36.661694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.661617 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" Oct 31 19:23:36.661694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.661648 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" event={"ID":"5276e29e-70b6-42e6-bdf2-f4f859ceaf08","Type":"ContainerDied","Data":"908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b"} Oct 31 19:23:36.661694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.661689 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg" event={"ID":"5276e29e-70b6-42e6-bdf2-f4f859ceaf08","Type":"ContainerDied","Data":"18f882abfaa56f8da7fc8beb2ea12373ffd6cdec47eafb5a7bb3c1876e91bc4a"} Oct 31 19:23:36.677349 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.677255 5688 scope.go:122] "RemoveContainer" containerID="351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b" Oct 31 19:23:36.677758 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:23:36.677728 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b\": container with ID starting with 351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b not found: ID does not exist" containerID="351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b" Oct 31 19:23:36.677886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.677762 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b"} err="failed to get container status \"351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b\": rpc error: code = NotFound desc = could not find container \"351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b\": container with ID starting with 351819b1b853c77a37aa86c19bc09dbe35172d54a2067a670f37ec833617fd9b not found: ID does not exist" Oct 31 19:23:36.677886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.677787 5688 scope.go:122] "RemoveContainer" containerID="908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b" Oct 31 19:23:36.687500 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.687485 5688 scope.go:122] "RemoveContainer" containerID="908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b" Oct 31 19:23:36.687791 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:23:36.687770 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b\": container with ID starting with 908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b not found: ID does not exist" containerID="908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b" Oct 31 19:23:36.687852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.687796 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b"} err="failed to get container status \"908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b\": rpc error: code = NotFound desc = could not find container \"908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b\": container with ID starting with 908800c9e0734b3ee449178c715ebd3e3886aace0ab32ab41bbce2bd60ee558b not found: ID does not exist" Oct 31 19:23:36.825544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.825519 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp"] Oct 31 19:23:36.834463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.834439 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8103d4-8d9b76d8c-hmnhp"] Oct 31 19:23:36.945089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.945014 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg"] Oct 31 19:23:36.947124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:36.947104 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8103d4-75d7c47785-h6djg"] Oct 31 19:23:37.567832 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:37.567803 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420222b8-86cc-4534-8323-7a6092b6e09f" path="/var/lib/kubelet/pods/420222b8-86cc-4534-8323-7a6092b6e09f/volumes" Oct 31 19:23:37.568261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:37.568245 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5276e29e-70b6-42e6-bdf2-f4f859ceaf08" path="/var/lib/kubelet/pods/5276e29e-70b6-42e6-bdf2-f4f859ceaf08/volumes" Oct 31 19:23:43.047433 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:43.047395 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz"] Oct 31 19:23:43.096760 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:43.096732 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d96a1c7-4sscz"] Oct 31 19:23:43.568102 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:23:43.568064 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf2d8e4-34b2-4992-8ff8-d8d646e2ec70" path="/var/lib/kubelet/pods/adf2d8e4-34b2-4992-8ff8-d8d646e2ec70/volumes" Oct 31 19:23:54.904646 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:23:54.904613 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:24:21.520700 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:21.520665 5688 scope.go:122] "RemoveContainer" containerID="79a61a4b6943474e5f0995d9e3a35447e8baa41490ce2be255117b7e11fe0d8d" Oct 31 19:24:21.529978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:21.529956 5688 scope.go:122] "RemoveContainer" containerID="7d3a122a042a45cb11c70553d1133567462cc5f24ccd512fb9edc967a2b2c8ee" Oct 31 19:24:24.904048 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:24:24.904007 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:24:42.625322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:42.625289 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh"] Oct 31 19:24:42.625749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:42.625491 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" podUID="1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" containerName="maas-api" containerID="cri-o://183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64" gracePeriod=30 Oct 31 19:24:42.936122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:42.936091 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:24:43.020205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.019861 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs\") pod \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " Oct 31 19:24:43.020205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.019915 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca\") pod \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " Oct 31 19:24:43.020205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.019957 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\") pod \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " Oct 31 19:24:43.020205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.020071 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj\" (UniqueName: \"kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj\") pod \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\" (UID: \"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c\") " Oct 31 19:24:43.027499 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.022879 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca" pod "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" (UID: "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:24:43.027499 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.026587 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls" pod "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" (UID: "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:24:43.031806 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.031765 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj" pod "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" (UID: "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c"). InnerVolumeSpecName "kube-api-access-t5mvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:24:43.033338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.033312 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs" pod "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" (UID: "1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:24:43.047235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.043234 5688 generic.go:356] "Generic (PLEG): container finished" podID="1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" containerID="183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64" exitCode=0 Oct 31 19:24:43.047235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.043445 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" event={"ID":"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c","Type":"ContainerDied","Data":"183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64"} Oct 31 19:24:43.047235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.043558 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" event={"ID":"1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c","Type":"ContainerDied","Data":"9d28a00c548106923dddf3f021b37e9bedd6bac110cb9a82376fbf2756f768fb"} Oct 31 19:24:43.047235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.043610 5688 scope.go:122] "RemoveContainer" containerID="183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64" Oct 31 19:24:43.047235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.043899 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh" Oct 31 19:24:43.067053 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.067026 5688 scope.go:122] "RemoveContainer" containerID="183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64" Oct 31 19:24:43.067475 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:24:43.067434 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64\": container with ID starting with 183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64 not found: ID does not exist" containerID="183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64" Oct 31 19:24:43.067572 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.067486 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64"} err="failed to get container status \"183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64\": rpc error: code = NotFound desc = could not find container \"183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64\": container with ID starting with 183a75921ccdbfee85dd24e29bb050877a196332d163fa1a41f121ea44465f64 not found: ID does not exist" Oct 31 19:24:43.121908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.121879 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mvj\" (UniqueName: \"kubernetes.io/projected/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-kube-api-access-t5mvj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:43.121908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.121904 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:43.122087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.121916 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:43.122087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.121926 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:43.216467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.216387 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh"] Oct 31 19:24:43.220970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.220941 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9cf479-694d85b98f-dz2lh"] Oct 31 19:24:43.565605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:43.565572 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" path="/var/lib/kubelet/pods/1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c/volumes" Oct 31 19:24:44.665623 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.665586 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5"] Oct 31 19:24:44.715269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.715237 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9cf479-l6lc5"] Oct 31 19:24:44.735085 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735059 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk"] Oct 31 19:24:44.735531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735518 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" containerName="maas-api" Oct 31 19:24:44.735581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735533 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" containerName="maas-api" Oct 31 19:24:44.735581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735553 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5276e29e-70b6-42e6-bdf2-f4f859ceaf08" containerName="payload-processing" Oct 31 19:24:44.735581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735558 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5276e29e-70b6-42e6-bdf2-f4f859ceaf08" containerName="payload-processing" Oct 31 19:24:44.735581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735568 5688 cpu_manager.go:414] "containerMap: removing container" podUID="420222b8-86cc-4534-8323-7a6092b6e09f" containerName="payload-pre-processing" Oct 31 19:24:44.735581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735573 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420222b8-86cc-4534-8323-7a6092b6e09f" containerName="payload-pre-processing" Oct 31 19:24:44.735774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735607 5688 cpu_manager.go:414] "containerMap: removing container" podUID="dda5ebe5-0759-485d-b224-92244204dd16" containerName="revoke-keys" Oct 31 19:24:44.735774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735612 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda5ebe5-0759-485d-b224-92244204dd16" containerName="revoke-keys" Oct 31 19:24:44.735774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735679 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4e8eb6-1b81-4012-8d8c-9fcc4cbaea5c" containerName="maas-api" Oct 31 19:24:44.735774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735688 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5276e29e-70b6-42e6-bdf2-f4f859ceaf08" containerName="payload-processing" Oct 31 19:24:44.735774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735696 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="420222b8-86cc-4534-8323-7a6092b6e09f" containerName="payload-pre-processing" Oct 31 19:24:44.735774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.735706 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda5ebe5-0759-485d-b224-92244204dd16" containerName="revoke-keys" Oct 31 19:24:44.740228 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.740209 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.793538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.793515 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:24:44.793682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.793656 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:24:44.796064 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.796044 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk"] Oct 31 19:24:44.842832 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.842802 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.843061 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.843042 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch4xg\" (UniqueName: \"kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.843344 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.843328 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.859078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.859049 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps"] Oct 31 19:24:44.863579 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.863499 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" containerID="cri-o://19ed436a4cd75ed4417bdb1a6ab325a46bc85fe345a458b80aedd1285db594bf" gracePeriod=30 Oct 31 19:24:44.945139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.945041 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.945139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.945130 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.945370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.945171 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch4xg\" (UniqueName: \"kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.956206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.946240 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:44.966207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:44.965461 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:45.004723 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:45.004223 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch4xg\" (UniqueName: \"kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg\") pod \"maas-api-revoke-sub-75b17f6d-8vpgk\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:45.052859 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:45.052832 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:45.428983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:45.428953 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk"] Oct 31 19:24:45.431004 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:24:45.430973 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46058ab_cd03_4f05_89fa_fad908227b1e.slice/crio-4e76dc86f6d2df497d443882abb699eeffe5b5e743f3e64c59d6c8214e187e41 WatchSource:0}: Error finding container 4e76dc86f6d2df497d443882abb699eeffe5b5e743f3e64c59d6c8214e187e41: Status 404 returned error can't find the container with id 4e76dc86f6d2df497d443882abb699eeffe5b5e743f3e64c59d6c8214e187e41 Oct 31 19:24:45.565674 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:45.565642 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda5ebe5-0759-485d-b224-92244204dd16" path="/var/lib/kubelet/pods/dda5ebe5-0759-485d-b224-92244204dd16/volumes" Oct 31 19:24:46.062839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:46.062761 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" event={"ID":"e46058ab-cd03-4f05-89fa-fad908227b1e","Type":"ContainerStarted","Data":"0c66a7f361b65d62fc88b3597ef5aef2aab2cf8d99f008397551cc5db57c506a"} Oct 31 19:24:46.062839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:46.062808 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" event={"ID":"e46058ab-cd03-4f05-89fa-fad908227b1e","Type":"ContainerStarted","Data":"4e76dc86f6d2df497d443882abb699eeffe5b5e743f3e64c59d6c8214e187e41"} Oct 31 19:24:46.181853 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:46.181808 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" podStartSLOduration=2.18179631 podStartE2EDuration="2.18179631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:46.181469958 +0000 UTC m=+3806.783611102" watchObservedRunningTime="2026-10-31 19:24:46.18179631 +0000 UTC m=+3806.783937455" Oct 31 19:24:47.080918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:47.080888 5688 generic.go:356] "Generic (PLEG): container finished" podID="e46058ab-cd03-4f05-89fa-fad908227b1e" containerID="0c66a7f361b65d62fc88b3597ef5aef2aab2cf8d99f008397551cc5db57c506a" exitCode=0 Oct 31 19:24:47.081330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:47.080978 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" event={"ID":"e46058ab-cd03-4f05-89fa-fad908227b1e","Type":"ContainerDied","Data":"0c66a7f361b65d62fc88b3597ef5aef2aab2cf8d99f008397551cc5db57c506a"} Oct 31 19:24:48.270764 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.270738 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:48.391304 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.387897 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca\") pod \"e46058ab-cd03-4f05-89fa-fad908227b1e\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " Oct 31 19:24:48.391304 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.388050 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca\") pod \"e46058ab-cd03-4f05-89fa-fad908227b1e\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " Oct 31 19:24:48.391304 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.388090 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg\" (UniqueName: \"kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg\") pod \"e46058ab-cd03-4f05-89fa-fad908227b1e\" (UID: \"e46058ab-cd03-4f05-89fa-fad908227b1e\") " Oct 31 19:24:48.393200 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.393138 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca" pod "e46058ab-cd03-4f05-89fa-fad908227b1e" (UID: "e46058ab-cd03-4f05-89fa-fad908227b1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:24:48.406974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.406312 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca" pod "e46058ab-cd03-4f05-89fa-fad908227b1e" (UID: "e46058ab-cd03-4f05-89fa-fad908227b1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:24:48.406974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.406354 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg" pod "e46058ab-cd03-4f05-89fa-fad908227b1e" (UID: "e46058ab-cd03-4f05-89fa-fad908227b1e"). InnerVolumeSpecName "kube-api-access-ch4xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:24:48.489967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.489934 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:48.489967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.489962 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46058ab-cd03-4f05-89fa-fad908227b1e-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:48.489967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.489971 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch4xg\" (UniqueName: \"kubernetes.io/projected/e46058ab-cd03-4f05-89fa-fad908227b1e-kube-api-access-ch4xg\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:48.801382 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.801346 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:24:48.801558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:48.801411 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:24:49.094686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:49.094604 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" event={"ID":"e46058ab-cd03-4f05-89fa-fad908227b1e","Type":"ContainerDied","Data":"4e76dc86f6d2df497d443882abb699eeffe5b5e743f3e64c59d6c8214e187e41"} Oct 31 19:24:49.094686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:49.094658 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e76dc86f6d2df497d443882abb699eeffe5b5e743f3e64c59d6c8214e187e41" Oct 31 19:24:49.094686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:49.094620 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk" Oct 31 19:24:50.103749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.103717 5688 generic.go:356] "Generic (PLEG): container finished" podID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerID="19ed436a4cd75ed4417bdb1a6ab325a46bc85fe345a458b80aedd1285db594bf" exitCode=0 Oct 31 19:24:50.104207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.103786 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" event={"ID":"7e49b6fc-0648-4079-9048-b24ca7dd097c","Type":"ContainerDied","Data":"19ed436a4cd75ed4417bdb1a6ab325a46bc85fe345a458b80aedd1285db594bf"} Oct 31 19:24:50.130895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.130840 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:24:50.208086 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208058 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208112 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208167 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208290 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208318 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208375 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208401 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208417 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:24:50.208543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208427 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:24:50.208543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208476 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208510 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket\") pod \"7e49b6fc-0648-4079-9048-b24ca7dd097c\" (UID: \"7e49b6fc-0648-4079-9048-b24ca7dd097c\") " Oct 31 19:24:50.208784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208620 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:24:50.208784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208763 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:24:50.208882 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208795 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:24:50.208962 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208945 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e49b6fc-0648-4079-9048-b24ca7dd097c-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.209022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208969 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.209022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.208985 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.209022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.209000 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.209022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.209014 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.210526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.210503 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:24:50.210836 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.210809 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:24:50.210934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.210874 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:24:50.210934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.210884 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn" pod "7e49b6fc-0648-4079-9048-b24ca7dd097c" (UID: "7e49b6fc-0648-4079-9048-b24ca7dd097c"). InnerVolumeSpecName "kube-api-access-pt9rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:24:50.310149 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.310113 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt9rn\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-kube-api-access-pt9rn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.310149 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.310146 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.310360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.310162 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.310360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.310171 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e49b6fc-0648-4079-9048-b24ca7dd097c-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:24:50.377980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.377946 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx"] Oct 31 19:24:50.378440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.378427 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e46058ab-cd03-4f05-89fa-fad908227b1e" containerName="revoke-keys" Oct 31 19:24:50.378495 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.378442 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46058ab-cd03-4f05-89fa-fad908227b1e" containerName="revoke-keys" Oct 31 19:24:50.378495 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.378463 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" Oct 31 19:24:50.378495 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.378468 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" Oct 31 19:24:50.378598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.378545 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46058ab-cd03-4f05-89fa-fad908227b1e" containerName="revoke-keys" Oct 31 19:24:50.378598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.378556 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" containerName="istio-proxy" Oct 31 19:24:50.382014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.381963 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.436047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.435995 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 19:24:50.436047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436024 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 19:24:50.436294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436117 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 19:24:50.436294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436137 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 19:24:50.436294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436172 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ce8c199032d2\"" Oct 31 19:24:50.436294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436215 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3b602a41f0e9\"" Oct 31 19:24:50.436440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436306 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c2025e74c640\"" Oct 31 19:24:50.436440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.436326 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qmk8p\"" Oct 31 19:24:50.442577 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.442556 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx"] Oct 31 19:24:50.512869 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.512841 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.512869 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.512869 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc24r\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.513026 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.513009 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.513145 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.513131 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.614236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.614166 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.614367 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.614337 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.614423 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.614409 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.614476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.614440 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc24r\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.615083 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.615062 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.615224 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.615135 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.616910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.616892 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.677586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.677534 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc24r\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r\") pod \"payload-processing-external-model-8448d5b966-n5xhx\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:50.695714 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:50.695687 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:51.070484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.070458 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx"] Oct 31 19:24:51.073461 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:24:51.073416 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bea685_ed65_4430_8284_9b91fa29c210.slice/crio-8131c42a447f514f873b90f3ac5feaf0a4f047430207a997a799e4019312982a WatchSource:0}: Error finding container 8131c42a447f514f873b90f3ac5feaf0a4f047430207a997a799e4019312982a: Status 404 returned error can't find the container with id 8131c42a447f514f873b90f3ac5feaf0a4f047430207a997a799e4019312982a Oct 31 19:24:51.143702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.143555 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" Oct 31 19:24:51.144269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.144238 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps" event={"ID":"7e49b6fc-0648-4079-9048-b24ca7dd097c","Type":"ContainerDied","Data":"492093189af2582dfd3e89be515231bea45affba8060a91d4f4d5a7188417a52"} Oct 31 19:24:51.144387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.144341 5688 scope.go:122] "RemoveContainer" containerID="19ed436a4cd75ed4417bdb1a6ab325a46bc85fe345a458b80aedd1285db594bf" Oct 31 19:24:51.146202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.146165 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" event={"ID":"43bea685-ed65-4430-8284-9b91fa29c210","Type":"ContainerStarted","Data":"8131c42a447f514f873b90f3ac5feaf0a4f047430207a997a799e4019312982a"} Oct 31 19:24:51.309837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.309801 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps"] Oct 31 19:24:51.314673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.314644 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-mnsps"] Oct 31 19:24:51.566041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:51.566018 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e49b6fc-0648-4079-9048-b24ca7dd097c" path="/var/lib/kubelet/pods/7e49b6fc-0648-4079-9048-b24ca7dd097c/volumes" Oct 31 19:24:52.155207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:52.155147 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" event={"ID":"43bea685-ed65-4430-8284-9b91fa29c210","Type":"ContainerStarted","Data":"0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32"} Oct 31 19:24:52.155590 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:52.155237 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:24:52.270746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:52.270695 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" podStartSLOduration=2.270679412 podStartE2EDuration="2.270679412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:52.269956164 +0000 UTC m=+3812.872097318" watchObservedRunningTime="2026-10-31 19:24:52.270679412 +0000 UTC m=+3812.872820557" Oct 31 19:24:54.905017 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:24:54.904985 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:24:55.495558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:24:55.495521 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx"] Oct 31 19:25:00.495903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.495825 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" podUID="43bea685-ed65-4430-8284-9b91fa29c210" containerName="payload-processing" containerID="cri-o://0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32" gracePeriod=25 Oct 31 19:25:00.633562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.633506 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m"] Oct 31 19:25:00.638748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.638724 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.710625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.709742 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m"] Oct 31 19:25:00.747745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.746135 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.747745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.746272 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.747745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.746444 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gh6\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.747745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.746493 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.848220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.845802 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:25:00.848220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.847833 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume\") pod \"43bea685-ed65-4430-8284-9b91fa29c210\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " Oct 31 19:25:00.848220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.847932 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r\") pod \"43bea685-ed65-4430-8284-9b91fa29c210\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " Oct 31 19:25:00.848220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.848007 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume\") pod \"43bea685-ed65-4430-8284-9b91fa29c210\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " Oct 31 19:25:00.848220 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.848073 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials\") pod \"43bea685-ed65-4430-8284-9b91fa29c210\" (UID: \"43bea685-ed65-4430-8284-9b91fa29c210\") " Oct 31 19:25:00.848597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.848379 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.848597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.848447 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.848597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.848503 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gh6\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.848597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.848533 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.849480 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.849450 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.850505 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.850476 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.850828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.850801 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume" pod "43bea685-ed65-4430-8284-9b91fa29c210" (UID: "43bea685-ed65-4430-8284-9b91fa29c210"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:00.851103 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.850994 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume" pod "43bea685-ed65-4430-8284-9b91fa29c210" (UID: "43bea685-ed65-4430-8284-9b91fa29c210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:00.857210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.852083 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.857210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.852315 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r" pod "43bea685-ed65-4430-8284-9b91fa29c210" (UID: "43bea685-ed65-4430-8284-9b91fa29c210"). InnerVolumeSpecName "kube-api-access-mc24r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:00.857210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.852340 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials" pod "43bea685-ed65-4430-8284-9b91fa29c210" (UID: "43bea685-ed65-4430-8284-9b91fa29c210"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:00.905904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.905877 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gh6\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6\") pod \"payload-processing-external-model-5c897f8c6f-bpx6m\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:00.949560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.949536 5688 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-routing-overlay-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:00.949560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.949564 5688 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-provider-credentials\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:00.949776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.949578 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bea685-ed65-4430-8284-9b91fa29c210-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:00.949776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.949591 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc24r\" (UniqueName: \"kubernetes.io/projected/43bea685-ed65-4430-8284-9b91fa29c210-kube-api-access-mc24r\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:00.958163 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:00.958137 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:01.226741 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.226702 5688 generic.go:356] "Generic (PLEG): container finished" podID="43bea685-ed65-4430-8284-9b91fa29c210" containerID="0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32" exitCode=0 Oct 31 19:25:01.226917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.226777 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" Oct 31 19:25:01.226917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.226786 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" event={"ID":"43bea685-ed65-4430-8284-9b91fa29c210","Type":"ContainerDied","Data":"0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32"} Oct 31 19:25:01.226917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.226885 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx" event={"ID":"43bea685-ed65-4430-8284-9b91fa29c210","Type":"ContainerDied","Data":"8131c42a447f514f873b90f3ac5feaf0a4f047430207a997a799e4019312982a"} Oct 31 19:25:01.226917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.226913 5688 scope.go:122] "RemoveContainer" containerID="0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32" Oct 31 19:25:01.238233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.238213 5688 scope.go:122] "RemoveContainer" containerID="0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32" Oct 31 19:25:01.238517 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:25:01.238497 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32\": container with ID starting with 0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32 not found: ID does not exist" containerID="0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32" Oct 31 19:25:01.238594 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.238526 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32"} err="failed to get container status \"0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32\": rpc error: code = NotFound desc = could not find container \"0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32\": container with ID starting with 0a8b1c3c7003d8deb84c1906efb933892242c2d57f83b7120bc3bcca0cb49b32 not found: ID does not exist" Oct 31 19:25:01.354649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.352778 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m"] Oct 31 19:25:01.375454 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:01.375419 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dbd0bf_6789_41f8_b91e_3dce83cea161.slice/crio-69b00a9fc09c8dbb8beef191030d16fdc09766ac23d6d34f82abac49a7877670 WatchSource:0}: Error finding container 69b00a9fc09c8dbb8beef191030d16fdc09766ac23d6d34f82abac49a7877670: Status 404 returned error can't find the container with id 69b00a9fc09c8dbb8beef191030d16fdc09766ac23d6d34f82abac49a7877670 Oct 31 19:25:01.404315 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.404280 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx"] Oct 31 19:25:01.404963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.404935 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8448d5b966-n5xhx"] Oct 31 19:25:01.571901 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:01.571862 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bea685-ed65-4430-8284-9b91fa29c210" path="/var/lib/kubelet/pods/43bea685-ed65-4430-8284-9b91fa29c210/volumes" Oct 31 19:25:02.254623 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:02.254581 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" event={"ID":"93dbd0bf-6789-41f8-b91e-3dce83cea161","Type":"ContainerStarted","Data":"2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89"} Oct 31 19:25:02.254926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:02.254901 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" event={"ID":"93dbd0bf-6789-41f8-b91e-3dce83cea161","Type":"ContainerStarted","Data":"69b00a9fc09c8dbb8beef191030d16fdc09766ac23d6d34f82abac49a7877670"} Oct 31 19:25:02.261054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:02.261026 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:03.148208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.141087 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" podStartSLOduration=3.141069328 podStartE2EDuration="3.141069328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:02.376249255 +0000 UTC m=+3822.978390401" watchObservedRunningTime="2026-10-31 19:25:03.141069328 +0000 UTC m=+3823.743210472" Oct 31 19:25:03.148208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.143974 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m"] Oct 31 19:25:03.175031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.175003 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z"] Oct 31 19:25:03.175485 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.175471 5688 cpu_manager.go:414] "containerMap: removing container" podUID="43bea685-ed65-4430-8284-9b91fa29c210" containerName="payload-processing" Oct 31 19:25:03.175540 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.175487 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bea685-ed65-4430-8284-9b91fa29c210" containerName="payload-processing" Oct 31 19:25:03.175601 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.175593 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bea685-ed65-4430-8284-9b91fa29c210" containerName="payload-processing" Oct 31 19:25:03.179063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.179049 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.194240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.194212 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.194354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.194302 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfg9w\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.194354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.194322 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.194481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.194425 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.231033 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.231011 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fc050d9d3c08\"" Oct 31 19:25:03.256395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.256366 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z"] Oct 31 19:25:03.295541 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.295469 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg9w\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.295541 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.295522 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.295805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.295616 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.295805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.295714 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.296970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.296510 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.296970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.296921 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.298606 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.298578 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.359659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.359624 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg9w\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w\") pod \"payload-processing-external-model-dfd4695f9-88b5z\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.493953 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.493921 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:03.929268 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:03.929244 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z"] Oct 31 19:25:03.936207 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:03.932164 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27dece5_1c0b_48bc_9b78_ada1cb645a4b.slice/crio-d0ee82ee1dabc951c3e738e01a1e87f02314e7f6fb99d66ec5be46f08e0acc85 WatchSource:0}: Error finding container d0ee82ee1dabc951c3e738e01a1e87f02314e7f6fb99d66ec5be46f08e0acc85: Status 404 returned error can't find the container with id d0ee82ee1dabc951c3e738e01a1e87f02314e7f6fb99d66ec5be46f08e0acc85 Oct 31 19:25:04.284001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:04.279996 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" event={"ID":"c27dece5-1c0b-48bc-9b78-ada1cb645a4b","Type":"ContainerStarted","Data":"615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6"} Oct 31 19:25:04.284001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:04.280123 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:04.284001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:04.280159 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" event={"ID":"c27dece5-1c0b-48bc-9b78-ada1cb645a4b","Type":"ContainerStarted","Data":"d0ee82ee1dabc951c3e738e01a1e87f02314e7f6fb99d66ec5be46f08e0acc85"} Oct 31 19:25:04.407963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:04.407909 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" podStartSLOduration=1.4078961190000001 podStartE2EDuration="1.407896119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:04.406509541 +0000 UTC m=+3825.008650685" watchObservedRunningTime="2026-10-31 19:25:04.407896119 +0000 UTC m=+3825.010037262" Oct 31 19:25:08.924588 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:08.924552 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm"] Oct 31 19:25:08.929354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:08.929338 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:08.985360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:08.985336 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c97ce8-dockercfg-hxxfx\"" Oct 31 19:25:09.009501 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.009374 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.009709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.009582 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh25\" (UniqueName: \"kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25\") pod \"payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.047227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.038461 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c97ce8\"" Oct 31 19:25:09.047227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.039565 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm"] Oct 31 19:25:09.110917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.110882 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.111098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.111009 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh25\" (UniqueName: \"kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25\") pod \"payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.111680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.111659 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.168044 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.168011 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh25\" (UniqueName: \"kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25\") pod \"payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.243174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.243145 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:09.279973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.279714 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" podUID="93dbd0bf-6789-41f8-b91e-3dce83cea161" containerName="payload-processing" containerID="cri-o://2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89" gracePeriod=25 Oct 31 19:25:09.562044 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.562018 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:09.676913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.676888 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm"] Oct 31 19:25:09.679554 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:09.679522 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda178472e_7915_49a6_b62e_bed8d86e3a54.slice/crio-561dddd2a982cd32d53220f94ed557e98aa0816a0c3ab0c222ace4d5c288cc1a WatchSource:0}: Error finding container 561dddd2a982cd32d53220f94ed557e98aa0816a0c3ab0c222ace4d5c288cc1a: Status 404 returned error can't find the container with id 561dddd2a982cd32d53220f94ed557e98aa0816a0c3ab0c222ace4d5c288cc1a Oct 31 19:25:09.726510 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.726483 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume\") pod \"93dbd0bf-6789-41f8-b91e-3dce83cea161\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " Oct 31 19:25:09.726665 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.726651 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume\") pod \"93dbd0bf-6789-41f8-b91e-3dce83cea161\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " Oct 31 19:25:09.726729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.726699 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials\") pod \"93dbd0bf-6789-41f8-b91e-3dce83cea161\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " Oct 31 19:25:09.726790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.726761 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6\") pod \"93dbd0bf-6789-41f8-b91e-3dce83cea161\" (UID: \"93dbd0bf-6789-41f8-b91e-3dce83cea161\") " Oct 31 19:25:09.726982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.726953 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume" pod "93dbd0bf-6789-41f8-b91e-3dce83cea161" (UID: "93dbd0bf-6789-41f8-b91e-3dce83cea161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:09.727141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.727023 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume" pod "93dbd0bf-6789-41f8-b91e-3dce83cea161" (UID: "93dbd0bf-6789-41f8-b91e-3dce83cea161"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:09.728904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.728883 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials" pod "93dbd0bf-6789-41f8-b91e-3dce83cea161" (UID: "93dbd0bf-6789-41f8-b91e-3dce83cea161"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:09.729022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.729004 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6" pod "93dbd0bf-6789-41f8-b91e-3dce83cea161" (UID: "93dbd0bf-6789-41f8-b91e-3dce83cea161"). InnerVolumeSpecName "kube-api-access-q8gh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:09.827816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.827740 5688 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-routing-overlay-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:09.827816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.827766 5688 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-provider-credentials\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:09.827816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.827776 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gh6\" (UniqueName: \"kubernetes.io/projected/93dbd0bf-6789-41f8-b91e-3dce83cea161-kube-api-access-q8gh6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:09.827816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:09.827785 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93dbd0bf-6789-41f8-b91e-3dce83cea161-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:10.320354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.320320 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" event={"ID":"a178472e-7915-49a6-b62e-bed8d86e3a54","Type":"ContainerStarted","Data":"010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b"} Oct 31 19:25:10.320787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.320373 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" event={"ID":"a178472e-7915-49a6-b62e-bed8d86e3a54","Type":"ContainerStarted","Data":"561dddd2a982cd32d53220f94ed557e98aa0816a0c3ab0c222ace4d5c288cc1a"} Oct 31 19:25:10.320787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.320420 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:10.321877 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.321853 5688 generic.go:356] "Generic (PLEG): container finished" podID="93dbd0bf-6789-41f8-b91e-3dce83cea161" containerID="2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89" exitCode=0 Oct 31 19:25:10.322019 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.321891 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" Oct 31 19:25:10.322019 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.321922 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" event={"ID":"93dbd0bf-6789-41f8-b91e-3dce83cea161","Type":"ContainerDied","Data":"2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89"} Oct 31 19:25:10.322019 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.321958 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m" event={"ID":"93dbd0bf-6789-41f8-b91e-3dce83cea161","Type":"ContainerDied","Data":"69b00a9fc09c8dbb8beef191030d16fdc09766ac23d6d34f82abac49a7877670"} Oct 31 19:25:10.322019 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.321979 5688 scope.go:122] "RemoveContainer" containerID="2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89" Oct 31 19:25:10.332133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.332112 5688 scope.go:122] "RemoveContainer" containerID="2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89" Oct 31 19:25:10.332478 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:25:10.332459 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89\": container with ID starting with 2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89 not found: ID does not exist" containerID="2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89" Oct 31 19:25:10.332562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.332490 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89"} err="failed to get container status \"2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89\": rpc error: code = NotFound desc = could not find container \"2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89\": container with ID starting with 2658636bac4f2bd90f4d9026514cbc65100c16bfeff8776659fa7d03cc9b9f89 not found: ID does not exist" Oct 31 19:25:10.436370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.436277 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" podStartSLOduration=2.436260703 podStartE2EDuration="2.436260703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:10.434809122 +0000 UTC m=+3831.036950265" watchObservedRunningTime="2026-10-31 19:25:10.436260703 +0000 UTC m=+3831.038401848" Oct 31 19:25:10.546958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.546927 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m"] Oct 31 19:25:10.550997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.550976 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-bpx6m"] Oct 31 19:25:10.777621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.777542 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z"] Oct 31 19:25:10.780621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.780594 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:10.802749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.802722 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6"] Oct 31 19:25:10.803399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.803378 5688 cpu_manager.go:414] "containerMap: removing container" podUID="93dbd0bf-6789-41f8-b91e-3dce83cea161" containerName="payload-processing" Oct 31 19:25:10.803399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.803401 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93dbd0bf-6789-41f8-b91e-3dce83cea161" containerName="payload-processing" Oct 31 19:25:10.803602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.803541 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="93dbd0bf-6789-41f8-b91e-3dce83cea161" containerName="payload-processing" Oct 31 19:25:10.806840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.806823 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:10.869386 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.869363 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6"] Oct 31 19:25:10.939203 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.939159 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79q7\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:10.939361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.939329 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:10.939403 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.939385 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:10.939450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:10.939437 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.040711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.040642 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.040711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.040689 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.040711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.040710 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.040916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.040753 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79q7\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.041413 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.041337 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.041413 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.041451 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.043498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.043468 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.098683 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.098649 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79q7\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7\") pod \"payload-processing-external-model-5c897f8c6f-2q9r6\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.120402 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.120382 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:11.330119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.330034 5688 generic.go:356] "Generic (PLEG): container finished" podID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerID="c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220" exitCode=0 Oct 31 19:25:11.330616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.330116 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" event={"ID":"7f134f9d-1862-48db-95d4-843d1a967dd9","Type":"ContainerDied","Data":"c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220"} Oct 31 19:25:11.330745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.330728 5688 scope.go:122] "RemoveContainer" containerID="c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220" Oct 31 19:25:11.332097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.332072 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:11.332163 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.332142 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:11.332227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.332176 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:11.504320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.500598 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6"] Oct 31 19:25:11.504320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.503811 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:25:11.505723 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:11.505679 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb3d398_3dbc_4319_979d_02f7d9b6946f.slice/crio-55124536877b529ecd1a9afc81d7733d1de4b9af259f80164b679d2bbb7d79e4 WatchSource:0}: Error finding container 55124536877b529ecd1a9afc81d7733d1de4b9af259f80164b679d2bbb7d79e4: Status 404 returned error can't find the container with id 55124536877b529ecd1a9afc81d7733d1de4b9af259f80164b679d2bbb7d79e4 Oct 31 19:25:11.573769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:11.573735 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93dbd0bf-6789-41f8-b91e-3dce83cea161" path="/var/lib/kubelet/pods/93dbd0bf-6789-41f8-b91e-3dce83cea161/volumes" Oct 31 19:25:12.356206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:12.355511 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" event={"ID":"9fb3d398-3dbc-4319-979d-02f7d9b6946f","Type":"ContainerStarted","Data":"da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec"} Oct 31 19:25:12.356206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:12.355583 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" event={"ID":"9fb3d398-3dbc-4319-979d-02f7d9b6946f","Type":"ContainerStarted","Data":"55124536877b529ecd1a9afc81d7733d1de4b9af259f80164b679d2bbb7d79e4"} Oct 31 19:25:12.356775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:12.356540 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:12.359232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:12.359204 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" event={"ID":"7f134f9d-1862-48db-95d4-843d1a967dd9","Type":"ContainerStarted","Data":"1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b"} Oct 31 19:25:12.490234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:12.490167 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" podStartSLOduration=2.49015073 podStartE2EDuration="2.49015073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:12.487274896 +0000 UTC m=+3833.089416040" watchObservedRunningTime="2026-10-31 19:25:12.49015073 +0000 UTC m=+3833.092291873" Oct 31 19:25:13.351209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:13.349354 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:25:13.351209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:13.349491 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:25:13.351209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:13.349533 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:25:14.354208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:14.353542 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:25:14.354208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:14.353602 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:25:15.351207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:15.345582 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:25:15.351207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:15.345643 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:25:15.652260 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:15.650147 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6"] Oct 31 19:25:15.778035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:15.777994 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" podUID="c27dece5-1c0b-48bc-9b78-ada1cb645a4b" containerName="payload-processing" containerID="cri-o://615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6" gracePeriod=25 Oct 31 19:25:16.072354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.072333 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:16.118257 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118232 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials\") pod \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " Oct 31 19:25:16.118388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118314 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume\") pod \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " Oct 31 19:25:16.118388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118346 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume\") pod \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " Oct 31 19:25:16.118491 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118461 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w\") pod \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\" (UID: \"c27dece5-1c0b-48bc-9b78-ada1cb645a4b\") " Oct 31 19:25:16.118704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118677 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume" pod "c27dece5-1c0b-48bc-9b78-ada1cb645a4b" (UID: "c27dece5-1c0b-48bc-9b78-ada1cb645a4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:16.118784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118736 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume" pod "c27dece5-1c0b-48bc-9b78-ada1cb645a4b" (UID: "c27dece5-1c0b-48bc-9b78-ada1cb645a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:16.118924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118880 5688 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-routing-overlay-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:16.118924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.118900 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:16.120498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.120478 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w" pod "c27dece5-1c0b-48bc-9b78-ada1cb645a4b" (UID: "c27dece5-1c0b-48bc-9b78-ada1cb645a4b"). InnerVolumeSpecName "kube-api-access-qfg9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:16.120585 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.120496 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials" pod "c27dece5-1c0b-48bc-9b78-ada1cb645a4b" (UID: "c27dece5-1c0b-48bc-9b78-ada1cb645a4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:16.219599 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.219563 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfg9w\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-kube-api-access-qfg9w\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:16.219599 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.219590 5688 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c27dece5-1c0b-48bc-9b78-ada1cb645a4b-provider-credentials\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:16.354571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.354526 5688 patch_prober.go:29] interesting pod/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:25:16.354781 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.354593 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:25:16.384830 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.384801 5688 generic.go:356] "Generic (PLEG): container finished" podID="c27dece5-1c0b-48bc-9b78-ada1cb645a4b" containerID="615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6" exitCode=0 Oct 31 19:25:16.385004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.384855 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" Oct 31 19:25:16.385004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.384876 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" event={"ID":"c27dece5-1c0b-48bc-9b78-ada1cb645a4b","Type":"ContainerDied","Data":"615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6"} Oct 31 19:25:16.385004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.384932 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z" event={"ID":"c27dece5-1c0b-48bc-9b78-ada1cb645a4b","Type":"ContainerDied","Data":"d0ee82ee1dabc951c3e738e01a1e87f02314e7f6fb99d66ec5be46f08e0acc85"} Oct 31 19:25:16.385004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.384955 5688 scope.go:122] "RemoveContainer" containerID="615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6" Oct 31 19:25:16.394950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.394930 5688 scope.go:122] "RemoveContainer" containerID="615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6" Oct 31 19:25:16.395340 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:25:16.395218 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6\": container with ID starting with 615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6 not found: ID does not exist" containerID="615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6" Oct 31 19:25:16.395340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.395252 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6"} err="failed to get container status \"615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6\": rpc error: code = NotFound desc = could not find container \"615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6\": container with ID starting with 615fef9e56dfead3e7f830143591aad1c16379f2b9cb78bbc6e761b84d742df6 not found: ID does not exist" Oct 31 19:25:16.568048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.568012 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z"] Oct 31 19:25:16.574050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:16.574025 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-88b5z"] Oct 31 19:25:17.375640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:17.375611 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:25:17.376336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:17.376228 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:25:17.376917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:17.376900 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:25:17.565722 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:17.565690 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27dece5-1c0b-48bc-9b78-ada1cb645a4b" path="/var/lib/kubelet/pods/c27dece5-1c0b-48bc-9b78-ada1cb645a4b/volumes" Oct 31 19:25:20.655391 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.655350 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" podUID="9fb3d398-3dbc-4319-979d-02f7d9b6946f" containerName="payload-processing" containerID="cri-o://da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec" gracePeriod=25 Oct 31 19:25:20.893064 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.893033 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr"] Oct 31 19:25:20.893620 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.893605 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c27dece5-1c0b-48bc-9b78-ada1cb645a4b" containerName="payload-processing" Oct 31 19:25:20.893686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.893622 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27dece5-1c0b-48bc-9b78-ada1cb645a4b" containerName="payload-processing" Oct 31 19:25:20.893734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.893722 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27dece5-1c0b-48bc-9b78-ada1cb645a4b" containerName="payload-processing" Oct 31 19:25:20.896989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.896973 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:20.940944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.940922 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:20.969069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969045 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials\") pod \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " Oct 31 19:25:20.969235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969087 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7\") pod \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " Oct 31 19:25:20.969235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969120 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume\") pod \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " Oct 31 19:25:20.969410 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969266 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume\") pod \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\" (UID: \"9fb3d398-3dbc-4319-979d-02f7d9b6946f\") " Oct 31 19:25:20.969527 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969505 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79qh\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:20.969593 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969578 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:20.969667 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969649 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:20.969743 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969597 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume" pod "9fb3d398-3dbc-4319-979d-02f7d9b6946f" (UID: "9fb3d398-3dbc-4319-979d-02f7d9b6946f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:20.969845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969796 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:20.969951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969842 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume" pod "9fb3d398-3dbc-4319-979d-02f7d9b6946f" (UID: "9fb3d398-3dbc-4319-979d-02f7d9b6946f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:25:20.969951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.969913 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:20.971385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.971358 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials" pod "9fb3d398-3dbc-4319-979d-02f7d9b6946f" (UID: "9fb3d398-3dbc-4319-979d-02f7d9b6946f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:20.971612 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:20.971592 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7" pod "9fb3d398-3dbc-4319-979d-02f7d9b6946f" (UID: "9fb3d398-3dbc-4319-979d-02f7d9b6946f"). InnerVolumeSpecName "kube-api-access-t79q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:21.004136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.004105 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr"] Oct 31 19:25:21.071115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071081 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.071312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071197 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.071312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071267 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79qh\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.071440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071343 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.071440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071402 5688 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fb3d398-3dbc-4319-979d-02f7d9b6946f-routing-overlay-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:21.071440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071419 5688 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-provider-credentials\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:21.071440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071433 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79q7\" (UniqueName: \"kubernetes.io/projected/9fb3d398-3dbc-4319-979d-02f7d9b6946f-kube-api-access-t79q7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:25:21.071892 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.071868 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.072235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.072211 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.074294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.074270 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.167252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.162631 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79qh\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh\") pod \"payload-processing-external-model-5c897f8c6f-lsxmr\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.237744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.237706 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:21.335269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.335246 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:25:21.419960 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.419872 5688 generic.go:356] "Generic (PLEG): container finished" podID="9fb3d398-3dbc-4319-979d-02f7d9b6946f" containerID="da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec" exitCode=0 Oct 31 19:25:21.420133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.419993 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" event={"ID":"9fb3d398-3dbc-4319-979d-02f7d9b6946f","Type":"ContainerDied","Data":"da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec"} Oct 31 19:25:21.420133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.420046 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" event={"ID":"9fb3d398-3dbc-4319-979d-02f7d9b6946f","Type":"ContainerDied","Data":"55124536877b529ecd1a9afc81d7733d1de4b9af259f80164b679d2bbb7d79e4"} Oct 31 19:25:21.420133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.420087 5688 scope.go:122] "RemoveContainer" containerID="da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec" Oct 31 19:25:21.420376 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.420283 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6" Oct 31 19:25:21.435604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.435585 5688 scope.go:122] "RemoveContainer" containerID="da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec" Oct 31 19:25:21.435900 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:25:21.435880 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec\": container with ID starting with da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec not found: ID does not exist" containerID="da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec" Oct 31 19:25:21.435964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.435908 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec"} err="failed to get container status \"da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec\": rpc error: code = NotFound desc = could not find container \"da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec\": container with ID starting with da440ae6b10e5937604511c1dc70a1e8ec05013a468a0764ed6bb30cb50d14ec not found: ID does not exist" Oct 31 19:25:21.566178 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.566146 5688 scope.go:122] "RemoveContainer" containerID="02586bd1b4cfcf7525052b1b735d8d9d6bff37fa08388c1004693f1efad32cc9" Oct 31 19:25:21.568255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.568228 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6"] Oct 31 19:25:21.569433 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.569412 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-2q9r6"] Oct 31 19:25:21.576276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.576251 5688 scope.go:122] "RemoveContainer" containerID="81e6010916e904e09da35c2c6e265ad881d232f6f3a0e807a51a50683a0bb2ec" Oct 31 19:25:21.682700 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:21.682674 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr"] Oct 31 19:25:21.684402 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:21.684377 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e95b725_8577_4237_be04_fb88f12b264c.slice/crio-c644148fd19663a27ace4819b14ea2597e764d631d275258ebfc3806140448d7 WatchSource:0}: Error finding container c644148fd19663a27ace4819b14ea2597e764d631d275258ebfc3806140448d7: Status 404 returned error can't find the container with id c644148fd19663a27ace4819b14ea2597e764d631d275258ebfc3806140448d7 Oct 31 19:25:22.426926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:22.426887 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" event={"ID":"7e95b725-8577-4237-be04-fb88f12b264c","Type":"ContainerStarted","Data":"56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742"} Oct 31 19:25:22.427131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:22.426945 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" event={"ID":"7e95b725-8577-4237-be04-fb88f12b264c","Type":"ContainerStarted","Data":"c644148fd19663a27ace4819b14ea2597e764d631d275258ebfc3806140448d7"} Oct 31 19:25:22.427131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:22.426993 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:25:22.545411 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:22.545361 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podStartSLOduration=2.545349495 podStartE2EDuration="2.545349495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:22.54328383 +0000 UTC m=+3843.145424974" watchObservedRunningTime="2026-10-31 19:25:22.545349495 +0000 UTC m=+3843.147490638" Oct 31 19:25:23.566503 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:23.566464 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb3d398-3dbc-4319-979d-02f7d9b6946f" path="/var/lib/kubelet/pods/9fb3d398-3dbc-4319-979d-02f7d9b6946f/volumes" Oct 31 19:25:24.907794 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:25:24.907764 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:25:29.185245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.185207 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7"] Oct 31 19:25:29.185921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.185904 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9fb3d398-3dbc-4319-979d-02f7d9b6946f" containerName="payload-processing" Oct 31 19:25:29.185973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.185925 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb3d398-3dbc-4319-979d-02f7d9b6946f" containerName="payload-processing" Oct 31 19:25:29.186129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.186118 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb3d398-3dbc-4319-979d-02f7d9b6946f" containerName="payload-processing" Oct 31 19:25:29.189626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.189611 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.242144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.242119 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ad2d15\"" Oct 31 19:25:29.244946 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.244922 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7"] Oct 31 19:25:29.300943 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.300906 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.301135 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.300995 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5s7\" (UniqueName: \"kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7\") pod \"payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.335449 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.335415 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ad2d15-dockercfg-52ffd\"" Oct 31 19:25:29.353974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.353938 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc"] Oct 31 19:25:29.357822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.357807 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.402154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.402124 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.402295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.402174 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5s7\" (UniqueName: \"kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7\") pod \"payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.402834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.402807 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.415380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.415359 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc"] Oct 31 19:25:29.459701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.459628 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5s7\" (UniqueName: \"kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7\") pod \"payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.503276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.503247 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2w72\" (UniqueName: \"kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72\") pod \"payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.503431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.503367 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:29.503497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.503421 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.604496 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.604466 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2w72\" (UniqueName: \"kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72\") pod \"payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.604680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.604562 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.605516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.605492 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.661470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.661435 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2w72\" (UniqueName: \"kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72\") pod \"payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.672796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.672769 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:29.877282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:29.877249 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7"] Oct 31 19:25:29.879662 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:29.879639 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fe3e83_3b53_4658_8e9e_3bb729ac7204.slice/crio-e82596714b4ba9e1267932860b2e8245254b76fc10e4a991c678ac1d4e3118db WatchSource:0}: Error finding container e82596714b4ba9e1267932860b2e8245254b76fc10e4a991c678ac1d4e3118db: Status 404 returned error can't find the container with id e82596714b4ba9e1267932860b2e8245254b76fc10e4a991c678ac1d4e3118db Oct 31 19:25:30.057728 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.057565 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc"] Oct 31 19:25:30.061035 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:30.061004 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566f2bea_79ab_4fd9_8d0d_e6ab246000c5.slice/crio-7144b6a3ddb49eb6b66ffec32f695e65fcbbc626465d01d8694b48a75662b980 WatchSource:0}: Error finding container 7144b6a3ddb49eb6b66ffec32f695e65fcbbc626465d01d8694b48a75662b980: Status 404 returned error can't find the container with id 7144b6a3ddb49eb6b66ffec32f695e65fcbbc626465d01d8694b48a75662b980 Oct 31 19:25:30.483931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.483884 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" event={"ID":"566f2bea-79ab-4fd9-8d0d-e6ab246000c5","Type":"ContainerStarted","Data":"913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697"} Oct 31 19:25:30.484384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.483994 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" event={"ID":"566f2bea-79ab-4fd9-8d0d-e6ab246000c5","Type":"ContainerStarted","Data":"7144b6a3ddb49eb6b66ffec32f695e65fcbbc626465d01d8694b48a75662b980"} Oct 31 19:25:30.484384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.484029 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:30.485580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.485556 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" event={"ID":"66fe3e83-3b53-4658-8e9e-3bb729ac7204","Type":"ContainerStarted","Data":"4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04"} Oct 31 19:25:30.485717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.485595 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" event={"ID":"66fe3e83-3b53-4658-8e9e-3bb729ac7204","Type":"ContainerStarted","Data":"e82596714b4ba9e1267932860b2e8245254b76fc10e4a991c678ac1d4e3118db"} Oct 31 19:25:30.485717 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.485661 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:30.602332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:30.602282 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" podStartSLOduration=1.602267462 podStartE2EDuration="1.602267462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:30.600164636 +0000 UTC m=+3851.202305762" watchObservedRunningTime="2026-10-31 19:25:30.602267462 +0000 UTC m=+3851.204408643" Oct 31 19:25:33.434487 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:33.434449 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:25:40.946058 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:40.928160 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" podStartSLOduration=11.928132494 podStartE2EDuration="11.928132494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:30.724327528 +0000 UTC m=+3851.326468674" watchObservedRunningTime="2026-10-31 19:25:40.928132494 +0000 UTC m=+3861.530273637" Oct 31 19:25:40.946058 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:40.929827 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85"] Oct 31 19:25:40.946058 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:40.937156 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:40.978624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:40.978590 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr"] Oct 31 19:25:40.983240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:40.983220 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.028440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.028410 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85"] Oct 31 19:25:41.037147 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037124 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037286 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037157 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037286 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037199 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037377 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037312 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppzp\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037426 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037367 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037436 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037516 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037568 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.037673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.037653 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.042229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.042205 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr"] Oct 31 19:25:41.138360 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138325 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138365 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138394 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138413 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138516 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138758 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138545 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138758 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138621 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138758 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138645 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138758 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138672 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138771 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138802 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138826 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138858 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppzp\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138881 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138927 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.138982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.138959 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.139544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139001 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.139544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139028 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56r2\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.139544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139170 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.139544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139217 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.139544 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139430 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.139830 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139586 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.139830 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.139652 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.140988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.140966 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.141312 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.141295 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.194992 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.194966 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.195146 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.195066 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppzp\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-m9q85\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.240146 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240120 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240156 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240262 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240291 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56r2\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240334 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240352 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240379 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240489 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240527 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240591 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240643 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.240955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240933 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.241027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240961 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.241027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.240987 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.242250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.242223 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.242867 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.242850 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.266929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.266910 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:41.297087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.297063 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56r2\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.297261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.297238 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6zdwr\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.494906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.494825 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:25:41.495081 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.494935 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:25:41.596253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.596223 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:41.696393 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.696334 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85"] Oct 31 19:25:41.700975 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:41.700943 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ea3028_c752_4997_8cff_5c0f8ad16fca.slice/crio-59e8d1fdca0cff9b0e90e1db9072baf0028d9fc9102d1b4506a2164c8e684c40 WatchSource:0}: Error finding container 59e8d1fdca0cff9b0e90e1db9072baf0028d9fc9102d1b4506a2164c8e684c40: Status 404 returned error can't find the container with id 59e8d1fdca0cff9b0e90e1db9072baf0028d9fc9102d1b4506a2164c8e684c40 Oct 31 19:25:41.703658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.703622 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:41.703751 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.703717 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:41.703797 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.703754 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:41.977924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.977883 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr"] Oct 31 19:25:41.980198 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:25:41.980152 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba88761_0d6b_4c65_a4a9_856b9d83a171.slice/crio-ef51a527f5a8549a1a8c0407fefb60b42da792cf3ef43e0c68fb08e2378aef83 WatchSource:0}: Error finding container ef51a527f5a8549a1a8c0407fefb60b42da792cf3ef43e0c68fb08e2378aef83: Status 404 returned error can't find the container with id ef51a527f5a8549a1a8c0407fefb60b42da792cf3ef43e0c68fb08e2378aef83 Oct 31 19:25:41.982761 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.982723 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:41.982860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.982800 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:41.982860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:41.982840 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:25:42.566726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.566691 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" event={"ID":"cba88761-0d6b-4c65-a4a9-856b9d83a171","Type":"ContainerStarted","Data":"303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52"} Oct 31 19:25:42.566944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.566752 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" event={"ID":"cba88761-0d6b-4c65-a4a9-856b9d83a171","Type":"ContainerStarted","Data":"ef51a527f5a8549a1a8c0407fefb60b42da792cf3ef43e0c68fb08e2378aef83"} Oct 31 19:25:42.568504 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.568472 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" event={"ID":"63ea3028-c752-4997-8cff-5c0f8ad16fca","Type":"ContainerStarted","Data":"bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117"} Oct 31 19:25:42.568628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.568509 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" event={"ID":"63ea3028-c752-4997-8cff-5c0f8ad16fca","Type":"ContainerStarted","Data":"59e8d1fdca0cff9b0e90e1db9072baf0028d9fc9102d1b4506a2164c8e684c40"} Oct 31 19:25:42.597326 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.597302 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:42.685799 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.685750 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podStartSLOduration=2.685734896 podStartE2EDuration="2.685734896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:42.682896425 +0000 UTC m=+3863.285037569" watchObservedRunningTime="2026-10-31 19:25:42.685734896 +0000 UTC m=+3863.287876112" Oct 31 19:25:42.807086 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:42.807032 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podStartSLOduration=2.80701528 podStartE2EDuration="2.80701528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:42.805769817 +0000 UTC m=+3863.407910960" watchObservedRunningTime="2026-10-31 19:25:42.80701528 +0000 UTC m=+3863.409156424" Oct 31 19:25:43.268283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:43.268235 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:43.270731 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:43.270697 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:43.270959 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:43.270928 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:43.476342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:43.476291 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:25:43.598712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:43.598638 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 19:25:43.598712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:43.598696 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 19:25:44.267559 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:44.267522 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:44.267847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:44.267819 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:44.597342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:44.597266 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 19:25:44.597342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:44.597312 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 19:25:45.281210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:45.275015 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:45.281210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:45.275072 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:45.596780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:45.596700 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 19:25:45.596780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:45.596750 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 19:25:46.273209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:46.267521 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:46.273209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:46.267580 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:46.597108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:46.597025 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 19:25:46.597108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:46.597090 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 19:25:47.270211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:47.267544 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:47.270211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:47.267602 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:47.598583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:47.598498 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 19:25:47.599175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:47.599131 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 19:25:48.267601 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:48.267567 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:48.267787 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:48.267613 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:48.596970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:48.596876 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 19:25:48.596970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:48.596924 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 19:25:49.268310 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:49.268277 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:25:49.268828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:49.268328 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:25:50.272297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:50.272266 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:50.272801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:50.272568 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:50.273239 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:50.273219 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:25:50.385876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:50.385848 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:50.386404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:50.386305 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:50.387138 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:50.387117 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:25:53.433781 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:25:53.433749 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:25:54.904215 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:25:54.904154 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:26:03.433900 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:03.433810 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:26:13.434376 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:13.434333 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:26:20.157537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:20.157507 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:26:20.159451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:20.159424 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:26:23.433568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:23.433533 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:26:24.943809 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:26:24.943768 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:26:33.433459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:33.433423 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:26:43.433818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:43.433781 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:26:49.567616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:49.567584 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:26:54.904528 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:26:54.904490 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:26:59.567990 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:26:59.567954 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:27:09.567662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:09.567626 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:27:19.567465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:19.567431 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:27:24.903908 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:27:24.903871 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:27:29.567554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:29.567478 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:27:37.401964 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.401924 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf"] Oct 31 19:27:37.406454 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.406427 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.510273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.510241 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c249cd-serving-cert\"" Oct 31 19:27:37.512850 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.512829 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf"] Oct 31 19:27:37.520705 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.520685 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.520784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.520717 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpp5b\" (UniqueName: \"kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.520784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.520749 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.520982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.520963 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.621836 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.621805 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.622019 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:27:37.621943 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-serving-cert: secret "maas-api-e2e-extmod-b-c249cd-serving-cert" not found Oct 31 19:27:37.622019 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:27:37.622011 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls podName:d8c6be1b-4020-4e57-97b7-95efa57d12fc nodeName:}" failed. No retries permitted until 2026-10-31 19:27:38.121991975 +0000 UTC m=+3978.724133114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls") pod "maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" (UID: "d8c6be1b-4020-4e57-97b7-95efa57d12fc") : secret "maas-api-e2e-extmod-b-c249cd-serving-cert" not found Oct 31 19:27:37.622162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.622033 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.622162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.622078 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp5b\" (UniqueName: \"kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.622162 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.622131 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.622745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.622723 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.624526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.624506 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:37.682691 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:37.682618 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp5b\" (UniqueName: \"kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:38.129535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:38.129500 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:38.131929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:38.131901 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:38.320793 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:38.320763 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:38.700221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:38.700163 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf"] Oct 31 19:27:38.702969 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:27:38.702943 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c6be1b_4020_4e57_97b7_95efa57d12fc.slice/crio-2bb79d57d41dbf34f8e5749c0c83ac6725b9db9fde58d8fcac83ea34fde75d90 WatchSource:0}: Error finding container 2bb79d57d41dbf34f8e5749c0c83ac6725b9db9fde58d8fcac83ea34fde75d90: Status 404 returned error can't find the container with id 2bb79d57d41dbf34f8e5749c0c83ac6725b9db9fde58d8fcac83ea34fde75d90 Oct 31 19:27:38.704533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:38.704513 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:27:39.250178 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:39.250142 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" event={"ID":"d8c6be1b-4020-4e57-97b7-95efa57d12fc","Type":"ContainerStarted","Data":"2bb79d57d41dbf34f8e5749c0c83ac6725b9db9fde58d8fcac83ea34fde75d90"} Oct 31 19:27:39.567643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:39.567607 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:27:40.257689 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:40.257651 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" event={"ID":"d8c6be1b-4020-4e57-97b7-95efa57d12fc","Type":"ContainerStarted","Data":"8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3"} Oct 31 19:27:40.258078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:40.257752 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:40.374650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:40.374603 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" podStartSLOduration=2.547605614 podStartE2EDuration="3.374591428s" totalImagesPullingTime="826.985814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:40.373732641 +0000 UTC m=+3980.975873785" watchObservedRunningTime="2026-10-31 19:27:40.374591428 +0000 UTC m=+3980.976732617" Oct 31 19:27:46.268796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:46.268770 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:27:49.570523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:49.570480 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:27:54.905611 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:27:54.905574 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:27:59.567536 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:27:59.567498 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:28:05.560568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:05.560533 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:28:15.560615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:15.560580 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:28:24.904625 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:28:24.904590 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:28:25.560498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:25.560462 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:28:31.283003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:31.282967 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc"] Oct 31 19:28:31.401860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:31.401829 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7"] Oct 31 19:28:35.560813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:35.560775 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:28:36.283684 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.283648 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" podUID="566f2bea-79ab-4fd9-8d0d-e6ab246000c5" containerName="payload-processing" containerID="cri-o://913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697" gracePeriod=25 Oct 31 19:28:36.403034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.402977 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" podUID="66fe3e83-3b53-4658-8e9e-3bb729ac7204" containerName="payload-pre-processing" containerID="cri-o://4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04" gracePeriod=25 Oct 31 19:28:36.536694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.536628 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:28:36.674374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.674343 5688 generic.go:356] "Generic (PLEG): container finished" podID="566f2bea-79ab-4fd9-8d0d-e6ab246000c5" containerID="913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697" exitCode=0 Oct 31 19:28:36.674873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.674441 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" Oct 31 19:28:36.674873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.674714 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:28:36.674873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.674752 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" event={"ID":"566f2bea-79ab-4fd9-8d0d-e6ab246000c5","Type":"ContainerDied","Data":"913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697"} Oct 31 19:28:36.674873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.674812 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc" event={"ID":"566f2bea-79ab-4fd9-8d0d-e6ab246000c5","Type":"ContainerDied","Data":"7144b6a3ddb49eb6b66ffec32f695e65fcbbc626465d01d8694b48a75662b980"} Oct 31 19:28:36.674873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.674841 5688 scope.go:122] "RemoveContainer" containerID="913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697" Oct 31 19:28:36.676012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.675994 5688 generic.go:356] "Generic (PLEG): container finished" podID="66fe3e83-3b53-4658-8e9e-3bb729ac7204" containerID="4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04" exitCode=0 Oct 31 19:28:36.676100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.676071 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" event={"ID":"66fe3e83-3b53-4658-8e9e-3bb729ac7204","Type":"ContainerDied","Data":"4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04"} Oct 31 19:28:36.676264 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.676100 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" event={"ID":"66fe3e83-3b53-4658-8e9e-3bb729ac7204","Type":"ContainerDied","Data":"e82596714b4ba9e1267932860b2e8245254b76fc10e4a991c678ac1d4e3118db"} Oct 31 19:28:36.685531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.685498 5688 scope.go:122] "RemoveContainer" containerID="913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697" Oct 31 19:28:36.685861 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:28:36.685837 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697\": container with ID starting with 913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697 not found: ID does not exist" containerID="913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697" Oct 31 19:28:36.685970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.685866 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697"} err="failed to get container status \"913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697\": rpc error: code = NotFound desc = could not find container \"913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697\": container with ID starting with 913e42b9c491feaf4b8681932a692f70a96da8212922fd86857ba248e1eca697 not found: ID does not exist" Oct 31 19:28:36.685970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.685881 5688 scope.go:122] "RemoveContainer" containerID="4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04" Oct 31 19:28:36.690541 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.690519 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume\") pod \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " Oct 31 19:28:36.690644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.690574 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72\" (UniqueName: \"kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72\") pod \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\" (UID: \"566f2bea-79ab-4fd9-8d0d-e6ab246000c5\") " Oct 31 19:28:36.690937 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.690903 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume" pod "566f2bea-79ab-4fd9-8d0d-e6ab246000c5" (UID: "566f2bea-79ab-4fd9-8d0d-e6ab246000c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:28:36.691045 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.690997 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:28:36.693211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.693166 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72" pod "566f2bea-79ab-4fd9-8d0d-e6ab246000c5" (UID: "566f2bea-79ab-4fd9-8d0d-e6ab246000c5"). InnerVolumeSpecName "kube-api-access-d2w72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:28:36.696905 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.696885 5688 scope.go:122] "RemoveContainer" containerID="4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04" Oct 31 19:28:36.697176 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:28:36.697150 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04\": container with ID starting with 4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04 not found: ID does not exist" containerID="4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04" Oct 31 19:28:36.697275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.697201 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04"} err="failed to get container status \"4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04\": rpc error: code = NotFound desc = could not find container \"4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04\": container with ID starting with 4068eb13843655ee68ce25f024b59711d584b0a4eaf1e4de20cb881edea95c04 not found: ID does not exist" Oct 31 19:28:36.791505 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.791454 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7\" (UniqueName: \"kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7\") pod \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " Oct 31 19:28:36.791624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.791525 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume\") pod \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\" (UID: \"66fe3e83-3b53-4658-8e9e-3bb729ac7204\") " Oct 31 19:28:36.791784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.791772 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2w72\" (UniqueName: \"kubernetes.io/projected/566f2bea-79ab-4fd9-8d0d-e6ab246000c5-kube-api-access-d2w72\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:28:36.791880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.791863 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume" pod "66fe3e83-3b53-4658-8e9e-3bb729ac7204" (UID: "66fe3e83-3b53-4658-8e9e-3bb729ac7204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:28:36.793519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.793500 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7" pod "66fe3e83-3b53-4658-8e9e-3bb729ac7204" (UID: "66fe3e83-3b53-4658-8e9e-3bb729ac7204"). InnerVolumeSpecName "kube-api-access-5h5s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:28:36.892554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.892523 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5s7\" (UniqueName: \"kubernetes.io/projected/66fe3e83-3b53-4658-8e9e-3bb729ac7204-kube-api-access-5h5s7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:28:36.892554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:36.892548 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fe3e83-3b53-4658-8e9e-3bb729ac7204-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:28:37.093164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:37.093085 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc"] Oct 31 19:28:37.096655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:37.096629 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad2d15-6c745f4dc-nv4mc"] Oct 31 19:28:37.565840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:37.565808 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566f2bea-79ab-4fd9-8d0d-e6ab246000c5" path="/var/lib/kubelet/pods/566f2bea-79ab-4fd9-8d0d-e6ab246000c5/volumes" Oct 31 19:28:37.683805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:37.683781 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7" Oct 31 19:28:37.847569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:37.847503 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7"] Oct 31 19:28:37.855119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:37.855088 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad2d15-6c749b596c-zr8n7"] Oct 31 19:28:39.566562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:39.566527 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fe3e83-3b53-4658-8e9e-3bb729ac7204" path="/var/lib/kubelet/pods/66fe3e83-3b53-4658-8e9e-3bb729ac7204/volumes" Oct 31 19:28:45.560547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:45.560508 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:28:54.904327 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:28:54.904292 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:28:55.560660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:28:55.560621 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:05.560431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:05.560353 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:15.560221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:15.560157 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:19.367018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:19.366980 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4"] Oct 31 19:29:19.470901 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:19.470875 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t"] Oct 31 19:29:21.674811 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:21.674782 5688 scope.go:122] "RemoveContainer" containerID="5babd879680216392f0d1bfbffecd098fd4f771bb4b2d7e854bbe9712a605a76" Oct 31 19:29:24.367523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.367483 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" podUID="676ae71a-496c-4164-9aeb-59ff3590afc3" containerName="payload-processing" containerID="cri-o://a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab" gracePeriod=25 Oct 31 19:29:24.471977 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.471922 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" podUID="9765f9b2-5824-425e-b2bb-7fb96955a640" containerName="payload-pre-processing" containerID="cri-o://a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6" gracePeriod=25 Oct 31 19:29:24.624686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.624627 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:29:24.740046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.740022 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume\") pod \"676ae71a-496c-4164-9aeb-59ff3590afc3\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " Oct 31 19:29:24.740153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.740073 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz\" (UniqueName: \"kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz\") pod \"676ae71a-496c-4164-9aeb-59ff3590afc3\" (UID: \"676ae71a-496c-4164-9aeb-59ff3590afc3\") " Oct 31 19:29:24.740405 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.740383 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume" pod "676ae71a-496c-4164-9aeb-59ff3590afc3" (UID: "676ae71a-496c-4164-9aeb-59ff3590afc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:29:24.740518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.740502 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ae71a-496c-4164-9aeb-59ff3590afc3-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:29:24.742372 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.742349 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz" pod "676ae71a-496c-4164-9aeb-59ff3590afc3" (UID: "676ae71a-496c-4164-9aeb-59ff3590afc3"). InnerVolumeSpecName "kube-api-access-8q8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:29:24.752397 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.752375 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:29:24.841055 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.841033 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume\") pod \"9765f9b2-5824-425e-b2bb-7fb96955a640\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " Oct 31 19:29:24.841247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.841230 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt\" (UniqueName: \"kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt\") pod \"9765f9b2-5824-425e-b2bb-7fb96955a640\" (UID: \"9765f9b2-5824-425e-b2bb-7fb96955a640\") " Oct 31 19:29:24.841449 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.841431 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume" pod "9765f9b2-5824-425e-b2bb-7fb96955a640" (UID: "9765f9b2-5824-425e-b2bb-7fb96955a640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:29:24.841621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.841601 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9765f9b2-5824-425e-b2bb-7fb96955a640-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:29:24.841621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.841623 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8xz\" (UniqueName: \"kubernetes.io/projected/676ae71a-496c-4164-9aeb-59ff3590afc3-kube-api-access-8q8xz\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:29:24.843406 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.843385 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt" pod "9765f9b2-5824-425e-b2bb-7fb96955a640" (UID: "9765f9b2-5824-425e-b2bb-7fb96955a640"). InnerVolumeSpecName "kube-api-access-828wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:29:24.905724 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:29:24.905648 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:29:24.942697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:24.942673 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828wt\" (UniqueName: \"kubernetes.io/projected/9765f9b2-5824-425e-b2bb-7fb96955a640-kube-api-access-828wt\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:29:25.022554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.022529 5688 generic.go:356] "Generic (PLEG): container finished" podID="9765f9b2-5824-425e-b2bb-7fb96955a640" containerID="a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6" exitCode=0 Oct 31 19:29:25.022706 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.022574 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" Oct 31 19:29:25.022706 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.022616 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" event={"ID":"9765f9b2-5824-425e-b2bb-7fb96955a640","Type":"ContainerDied","Data":"a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6"} Oct 31 19:29:25.022706 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.022662 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t" event={"ID":"9765f9b2-5824-425e-b2bb-7fb96955a640","Type":"ContainerDied","Data":"e60bac708c51acaf33dbd9f87633d5402bb680d1041d02e4774ec8ed6b6a5a90"} Oct 31 19:29:25.022706 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.022683 5688 scope.go:122] "RemoveContainer" containerID="a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6" Oct 31 19:29:25.024106 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.024087 5688 generic.go:356] "Generic (PLEG): container finished" podID="676ae71a-496c-4164-9aeb-59ff3590afc3" containerID="a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab" exitCode=0 Oct 31 19:29:25.024242 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.024140 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" Oct 31 19:29:25.024242 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.024163 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" event={"ID":"676ae71a-496c-4164-9aeb-59ff3590afc3","Type":"ContainerDied","Data":"a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab"} Oct 31 19:29:25.024332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.024245 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4" event={"ID":"676ae71a-496c-4164-9aeb-59ff3590afc3","Type":"ContainerDied","Data":"d024455549d6cfde1c29cbaed88c509cc854515eb03d817f3392f61245dd7bc4"} Oct 31 19:29:25.032318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.032300 5688 scope.go:122] "RemoveContainer" containerID="a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6" Oct 31 19:29:25.032651 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:29:25.032614 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6\": container with ID starting with a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6 not found: ID does not exist" containerID="a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6" Oct 31 19:29:25.032736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.032647 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6"} err="failed to get container status \"a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6\": rpc error: code = NotFound desc = could not find container \"a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6\": container with ID starting with a6622be9a223957119d30089ebcf23c537b2a204e3815cc34e28383e6c1186f6 not found: ID does not exist" Oct 31 19:29:25.032736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.032669 5688 scope.go:122] "RemoveContainer" containerID="a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab" Oct 31 19:29:25.041723 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.041707 5688 scope.go:122] "RemoveContainer" containerID="a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab" Oct 31 19:29:25.042002 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:29:25.041979 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab\": container with ID starting with a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab not found: ID does not exist" containerID="a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab" Oct 31 19:29:25.042099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.042007 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab"} err="failed to get container status \"a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab\": rpc error: code = NotFound desc = could not find container \"a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab\": container with ID starting with a760df6ffe91105341a7a9fbc909f010f1f22e0c33eaa8bf8e3a8aa0d9f893ab not found: ID does not exist" Oct 31 19:29:25.187048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.186976 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t"] Oct 31 19:29:25.190130 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.190107 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b70248-65b6788dc5-9wk4t"] Oct 31 19:29:25.298830 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.298802 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4"] Oct 31 19:29:25.301857 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.301831 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b70248-58595b797c-xtbp4"] Oct 31 19:29:25.560460 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.560425 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:25.566645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.566622 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676ae71a-496c-4164-9aeb-59ff3590afc3" path="/var/lib/kubelet/pods/676ae71a-496c-4164-9aeb-59ff3590afc3/volumes" Oct 31 19:29:25.566984 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:25.566971 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9765f9b2-5824-425e-b2bb-7fb96955a640" path="/var/lib/kubelet/pods/9765f9b2-5824-425e-b2bb-7fb96955a640/volumes" Oct 31 19:29:32.594989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.594949 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth"] Oct 31 19:29:32.595481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595467 5688 cpu_manager.go:414] "containerMap: removing container" podUID="66fe3e83-3b53-4658-8e9e-3bb729ac7204" containerName="payload-pre-processing" Oct 31 19:29:32.595528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595482 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fe3e83-3b53-4658-8e9e-3bb729ac7204" containerName="payload-pre-processing" Oct 31 19:29:32.595528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595502 5688 cpu_manager.go:414] "containerMap: removing container" podUID="566f2bea-79ab-4fd9-8d0d-e6ab246000c5" containerName="payload-processing" Oct 31 19:29:32.595528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595507 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566f2bea-79ab-4fd9-8d0d-e6ab246000c5" containerName="payload-processing" Oct 31 19:29:32.595528 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595527 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9765f9b2-5824-425e-b2bb-7fb96955a640" containerName="payload-pre-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595531 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9765f9b2-5824-425e-b2bb-7fb96955a640" containerName="payload-pre-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595538 5688 cpu_manager.go:414] "containerMap: removing container" podUID="676ae71a-496c-4164-9aeb-59ff3590afc3" containerName="payload-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595543 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676ae71a-496c-4164-9aeb-59ff3590afc3" containerName="payload-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595619 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fe3e83-3b53-4658-8e9e-3bb729ac7204" containerName="payload-pre-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595632 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="566f2bea-79ab-4fd9-8d0d-e6ab246000c5" containerName="payload-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595641 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9765f9b2-5824-425e-b2bb-7fb96955a640" containerName="payload-pre-processing" Oct 31 19:29:32.595654 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.595646 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="676ae71a-496c-4164-9aeb-59ff3590afc3" containerName="payload-processing" Oct 31 19:29:32.600247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.600230 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.653767 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.653738 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fdd317-serving-cert\"" Oct 31 19:29:32.705804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.705773 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth"] Oct 31 19:29:32.715021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.714975 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.715167 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.715062 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6gs6\" (UniqueName: \"kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.715280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.715262 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.715345 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.715305 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.816417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.816384 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.816417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.816419 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6gs6\" (UniqueName: \"kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.816574 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.816478 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.816574 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.816498 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.816666 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:29:32.816648 5688 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-serving-cert: secret "maas-api-e2e-inf-b-fdd317-serving-cert" not found Oct 31 19:29:32.816723 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:29:32.816712 5688 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls podName:15db64a1-85fd-48ee-a80c-391fd786d58d nodeName:}" failed. No retries permitted until 2026-10-31 19:29:33.316692208 +0000 UTC m=+4093.918833345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls") pod "maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" (UID: "15db64a1-85fd-48ee-a80c-391fd786d58d") : secret "maas-api-e2e-inf-b-fdd317-serving-cert" not found Oct 31 19:29:32.817179 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.817158 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.818820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.818797 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:32.877249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:32.876633 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6gs6\" (UniqueName: \"kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:33.325956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:33.325927 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:33.328463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:33.328436 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:33.512791 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:33.512760 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:33.887589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:33.887567 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth"] Oct 31 19:29:33.889901 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:29:33.889871 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15db64a1_85fd_48ee_a80c_391fd786d58d.slice/crio-c1cc046db4c8cfea8a4bd315c3c9ef7b03255ece9c147899a1376a6eb4feab05 WatchSource:0}: Error finding container c1cc046db4c8cfea8a4bd315c3c9ef7b03255ece9c147899a1376a6eb4feab05: Status 404 returned error can't find the container with id c1cc046db4c8cfea8a4bd315c3c9ef7b03255ece9c147899a1376a6eb4feab05 Oct 31 19:29:34.075247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:34.075218 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" event={"ID":"15db64a1-85fd-48ee-a80c-391fd786d58d","Type":"ContainerStarted","Data":"c1cc046db4c8cfea8a4bd315c3c9ef7b03255ece9c147899a1376a6eb4feab05"} Oct 31 19:29:35.082858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:35.082773 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" event={"ID":"15db64a1-85fd-48ee-a80c-391fd786d58d","Type":"ContainerStarted","Data":"b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb"} Oct 31 19:29:35.082858 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:35.082836 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:35.201114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:35.201064 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" podStartSLOduration=2.30298924 podStartE2EDuration="3.201046121s" totalImagesPullingTime="898.056881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:35.198425775 +0000 UTC m=+4095.800566918" watchObservedRunningTime="2026-10-31 19:29:35.201046121 +0000 UTC m=+4095.803187270" Oct 31 19:29:35.560357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:35.560322 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:41.093571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:41.093544 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:29:45.561013 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:45.560978 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:49.048400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:49.048367 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk"] Oct 31 19:29:49.098238 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:49.098201 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b17f6d-8vpgk"] Oct 31 19:29:49.567128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:49.567096 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46058ab-cd03-4f05-89fa-fad908227b1e" path="/var/lib/kubelet/pods/e46058ab-cd03-4f05-89fa-fad908227b1e/volumes" Oct 31 19:29:54.905859 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:29:54.905811 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:29:55.560204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:55.560145 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:29:59.353156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:59.353115 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr"] Oct 31 19:29:59.355298 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:29:59.355267 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:30:00.179748 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.179718 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc"] Oct 31 19:30:00.183752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.183735 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd"] Oct 31 19:30:00.183863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.183841 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.187370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.187351 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.226238 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.226205 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr"] Oct 31 19:30:00.249728 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.237372 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs"] Oct 31 19:30:00.249728 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.239349 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.249728 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.240581 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:30:00.266942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.266918 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz"] Oct 31 19:30:00.267244 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.267000 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.272369 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.272346 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.277733 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.277704 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx"] Oct 31 19:30:00.288270 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.284328 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8"] Oct 31 19:30:00.294272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.290544 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.294272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.290764 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.316752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.315153 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc"] Oct 31 19:30:00.324424 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.324399 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr"] Oct 31 19:30:00.339258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.339231 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs"] Oct 31 19:30:00.351289 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.351251 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz"] Oct 31 19:30:00.355205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.355167 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx"] Oct 31 19:30:00.363420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.363388 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8"] Oct 31 19:30:00.401114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.401076 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrn6\" (UniqueName: \"kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.401523 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.401491 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbqw\" (UniqueName: \"kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw\") pod \"maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.401740 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.401720 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.401904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.401890 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwsm\" (UniqueName: \"kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.401995 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.401983 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56fg\" (UniqueName: \"kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg\") pod \"maas-api-key-cleanup-29891250-f8cxd\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.402099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.402088 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.402202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.402170 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891250-f8cxd\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.402315 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.402304 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.403508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.403485 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd"] Oct 31 19:30:00.503583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.503368 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbqw\" (UniqueName: \"kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw\") pod \"maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.504004 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdz6\" (UniqueName: \"kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.504218 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.504288 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrv8\" (UniqueName: \"kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505065 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505165 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwsm\" (UniqueName: \"kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505234 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56fg\" (UniqueName: \"kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg\") pod \"maas-api-key-cleanup-29891250-f8cxd\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505286 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505321 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891250-f8cxd\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505366 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505420 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrn6\" (UniqueName: \"kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505458 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505555 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fv6t\" (UniqueName: \"kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t\") pod \"maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.505610 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.507215 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.506616 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.508002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.507868 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.508619 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.508593 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891250-f8cxd\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.509500 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.509477 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.586167 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.586099 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56fg\" (UniqueName: \"kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg\") pod \"maas-api-key-cleanup-29891250-f8cxd\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.586378 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.586243 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwsm\" (UniqueName: \"kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.587712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.587683 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrn6\" (UniqueName: \"kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.588291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.588269 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbqw\" (UniqueName: \"kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw\") pod \"maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:00.606865 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.606839 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fv6t\" (UniqueName: \"kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t\") pod \"maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.606992 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.606890 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.606992 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.606960 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdz6\" (UniqueName: \"kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.606992 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.606988 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.607203 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.607151 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrv8\" (UniqueName: \"kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.607362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.607341 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.607624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.607606 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.607702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.607646 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.608141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.608111 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.641322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.641297 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:00.664271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.664246 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fv6t\" (UniqueName: \"kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t\") pod \"maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.664488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.664459 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrv8\" (UniqueName: \"kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.666080 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.666051 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdz6\" (UniqueName: \"kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.731868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.729997 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:00.743039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.743013 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:00.756081 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.756058 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:00.838805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.836592 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:00.841844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.840208 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:00.878863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:00.878652 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:01.034976 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.034860 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs"] Oct 31 19:30:01.170856 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.170832 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz"] Oct 31 19:30:01.173363 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:30:01.173333 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0980939e_1112_4300_91d8_a4e9437df583.slice/crio-e6ed873718dfd18964fa2ac268a15e24d8fe53309b6901f0accd3c69d4c9b223 WatchSource:0}: Error finding container e6ed873718dfd18964fa2ac268a15e24d8fe53309b6901f0accd3c69d4c9b223: Status 404 returned error can't find the container with id e6ed873718dfd18964fa2ac268a15e24d8fe53309b6901f0accd3c69d4c9b223 Oct 31 19:30:01.215517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.215494 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx"] Oct 31 19:30:01.218334 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:30:01.218230 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59eba41_82be_4721_8082_212b6df76bed.slice/crio-4d85d61e54206c6e77f5153b1d7ab5453a3e1731840c16427f553f0e232596f7 WatchSource:0}: Error finding container 4d85d61e54206c6e77f5153b1d7ab5453a3e1731840c16427f553f0e232596f7: Status 404 returned error can't find the container with id 4d85d61e54206c6e77f5153b1d7ab5453a3e1731840c16427f553f0e232596f7 Oct 31 19:30:01.310657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.310628 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerStarted","Data":"f455d384e1b33d1ed83e643ca9247969dbb75cc2984bc049d768a86c2ed39c25"} Oct 31 19:30:01.312094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.312070 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerStarted","Data":"4d85d61e54206c6e77f5153b1d7ab5453a3e1731840c16427f553f0e232596f7"} Oct 31 19:30:01.313581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.313555 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerStarted","Data":"e6ed873718dfd18964fa2ac268a15e24d8fe53309b6901f0accd3c69d4c9b223"} Oct 31 19:30:01.325297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.324818 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr"] Oct 31 19:30:01.328315 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.328291 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc"] Oct 31 19:30:01.334954 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:30:01.334930 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c7051a_1c9b_457b_9bde_b83e3878f2c2.slice/crio-226d061a527edd90dec5d8f19b80493526471f0cbaf72fa61217a9e06fa33f30 WatchSource:0}: Error finding container 226d061a527edd90dec5d8f19b80493526471f0cbaf72fa61217a9e06fa33f30: Status 404 returned error can't find the container with id 226d061a527edd90dec5d8f19b80493526471f0cbaf72fa61217a9e06fa33f30 Oct 31 19:30:01.336668 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:30:01.336644 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2f39c2_09dc_4648_803d_9fd91d3b091d.slice/crio-38eb24b18d0d6356084496e9c90daa0a19d50a56e74e191f977d8fe7bf378c19 WatchSource:0}: Error finding container 38eb24b18d0d6356084496e9c90daa0a19d50a56e74e191f977d8fe7bf378c19: Status 404 returned error can't find the container with id 38eb24b18d0d6356084496e9c90daa0a19d50a56e74e191f977d8fe7bf378c19 Oct 31 19:30:01.347919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.347893 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd"] Oct 31 19:30:01.357852 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:30:01.357811 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0c8e9d_ef25_4ecc_b8d0_0b2b12fc6de1.slice/crio-ec58451537d259d5a6eebb8a4d1ae1821d73219a10f7c9f18454312f80771f8b WatchSource:0}: Error finding container ec58451537d259d5a6eebb8a4d1ae1821d73219a10f7c9f18454312f80771f8b: Status 404 returned error can't find the container with id ec58451537d259d5a6eebb8a4d1ae1821d73219a10f7c9f18454312f80771f8b Oct 31 19:30:01.379732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:01.370814 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8"] Oct 31 19:30:01.379732 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:30:01.376952 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87498025_815e_4cec_b6db_a79423447931.slice/crio-a5b8d04eceefbaac0de7e0873942fc1cc720caabf49907dd233d8b4f9db03084 WatchSource:0}: Error finding container a5b8d04eceefbaac0de7e0873942fc1cc720caabf49907dd233d8b4f9db03084: Status 404 returned error can't find the container with id a5b8d04eceefbaac0de7e0873942fc1cc720caabf49907dd233d8b4f9db03084 Oct 31 19:30:02.326385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.326349 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b51bfe4-eaad-401d-924e-636f65470576" containerID="fa000aff302e16500b3de676ae84a981e66aa5732b8f6ba968cc98e6080f1055" exitCode=60 Oct 31 19:30:02.326647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.326459 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerDied","Data":"fa000aff302e16500b3de676ae84a981e66aa5732b8f6ba968cc98e6080f1055"} Oct 31 19:30:02.326887 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.326866 5688 scope.go:122] "RemoveContainer" containerID="fa000aff302e16500b3de676ae84a981e66aa5732b8f6ba968cc98e6080f1055" Oct 31 19:30:02.330292 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.329546 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerStarted","Data":"d560146bfa9bcdcede355b9c8fad49f4e2f28a55b51efb71821fee782beff98e"} Oct 31 19:30:02.330292 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.329599 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerStarted","Data":"38eb24b18d0d6356084496e9c90daa0a19d50a56e74e191f977d8fe7bf378c19"} Oct 31 19:30:02.372478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.367799 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerStarted","Data":"f8bb45cecc1b10e8c9a8087a589f755a55c8ed4b8b27690a2ad77cd670ab934c"} Oct 31 19:30:02.372478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.367860 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerStarted","Data":"ec58451537d259d5a6eebb8a4d1ae1821d73219a10f7c9f18454312f80771f8b"} Oct 31 19:30:02.372478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.370969 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerStarted","Data":"aa312709201cbb5e8c7a9391a3f6902fa1e5b699a9ddcc847ebe37e4bbb05735"} Oct 31 19:30:02.372478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.371020 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerStarted","Data":"a5b8d04eceefbaac0de7e0873942fc1cc720caabf49907dd233d8b4f9db03084"} Oct 31 19:30:02.388265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.387157 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerStarted","Data":"e7f916a1fbf51f3dba7a8cec0351f2f8d46860f63118cf6b9711eb08d4cc75a4"} Oct 31 19:30:02.390208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.389122 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerStarted","Data":"3d01987b46b29998e3babeb661381e6903cd95cbe7893adbfc83e3c67faa1067"} Oct 31 19:30:02.390208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.389442 5688 scope.go:122] "RemoveContainer" containerID="3d01987b46b29998e3babeb661381e6903cd95cbe7893adbfc83e3c67faa1067" Oct 31 19:30:02.391463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.391387 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerStarted","Data":"c3cb81e7e522bdcc4cf188ba42220d2f8dde3f76dd151c61e0db38ae294c8dc0"} Oct 31 19:30:02.391463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.391431 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerStarted","Data":"226d061a527edd90dec5d8f19b80493526471f0cbaf72fa61217a9e06fa33f30"} Oct 31 19:30:02.715457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.714965 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" podStartSLOduration=2.714925569 podStartE2EDuration="2.714925569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.575658885 +0000 UTC m=+4123.177800031" watchObservedRunningTime="2026-10-31 19:30:02.714925569 +0000 UTC m=+4123.317066713" Oct 31 19:30:02.716805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:02.715990 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" podStartSLOduration=2.715975211 podStartE2EDuration="2.715975211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.690951855 +0000 UTC m=+4123.293093000" watchObservedRunningTime="2026-10-31 19:30:02.715975211 +0000 UTC m=+4123.318116355" Oct 31 19:30:03.040504 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.040440 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" podStartSLOduration=3.04042022 podStartE2EDuration="3.04042022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.921320205 +0000 UTC m=+4123.523461349" watchObservedRunningTime="2026-10-31 19:30:03.04042022 +0000 UTC m=+4123.642561366" Oct 31 19:30:03.043272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.043223 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" podStartSLOduration=3.043205181 podStartE2EDuration="3.043205181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.040791741 +0000 UTC m=+4123.642932887" watchObservedRunningTime="2026-10-31 19:30:03.043205181 +0000 UTC m=+4123.645346321" Oct 31 19:30:03.156684 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.156641 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" podStartSLOduration=3.156626932 podStartE2EDuration="3.156626932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.154547528 +0000 UTC m=+4123.756688673" watchObservedRunningTime="2026-10-31 19:30:03.156626932 +0000 UTC m=+4123.758768075" Oct 31 19:30:03.402710 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.402674 5688 generic.go:356] "Generic (PLEG): container finished" podID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerID="c3cb81e7e522bdcc4cf188ba42220d2f8dde3f76dd151c61e0db38ae294c8dc0" exitCode=60 Oct 31 19:30:03.403105 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.402758 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerDied","Data":"c3cb81e7e522bdcc4cf188ba42220d2f8dde3f76dd151c61e0db38ae294c8dc0"} Oct 31 19:30:03.403150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.403126 5688 scope.go:122] "RemoveContainer" containerID="c3cb81e7e522bdcc4cf188ba42220d2f8dde3f76dd151c61e0db38ae294c8dc0" Oct 31 19:30:03.405330 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.405306 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerStarted","Data":"b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64"} Oct 31 19:30:03.407259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.407227 5688 generic.go:356] "Generic (PLEG): container finished" podID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerID="d560146bfa9bcdcede355b9c8fad49f4e2f28a55b51efb71821fee782beff98e" exitCode=60 Oct 31 19:30:03.407381 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.407355 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerDied","Data":"d560146bfa9bcdcede355b9c8fad49f4e2f28a55b51efb71821fee782beff98e"} Oct 31 19:30:03.407535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.407520 5688 scope.go:122] "RemoveContainer" containerID="d560146bfa9bcdcede355b9c8fad49f4e2f28a55b51efb71821fee782beff98e" Oct 31 19:30:03.409450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.409412 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerID="f8bb45cecc1b10e8c9a8087a589f755a55c8ed4b8b27690a2ad77cd670ab934c" exitCode=60 Oct 31 19:30:03.409607 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.409528 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerDied","Data":"f8bb45cecc1b10e8c9a8087a589f755a55c8ed4b8b27690a2ad77cd670ab934c"} Oct 31 19:30:03.409715 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.409695 5688 scope.go:122] "RemoveContainer" containerID="f8bb45cecc1b10e8c9a8087a589f755a55c8ed4b8b27690a2ad77cd670ab934c" Oct 31 19:30:03.411311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.411290 5688 generic.go:356] "Generic (PLEG): container finished" podID="87498025-815e-4cec-b6db-a79423447931" containerID="aa312709201cbb5e8c7a9391a3f6902fa1e5b699a9ddcc847ebe37e4bbb05735" exitCode=60 Oct 31 19:30:03.411562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.411533 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerDied","Data":"aa312709201cbb5e8c7a9391a3f6902fa1e5b699a9ddcc847ebe37e4bbb05735"} Oct 31 19:30:03.411697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.411683 5688 scope.go:122] "RemoveContainer" containerID="aa312709201cbb5e8c7a9391a3f6902fa1e5b699a9ddcc847ebe37e4bbb05735" Oct 31 19:30:03.413165 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.413141 5688 generic.go:356] "Generic (PLEG): container finished" podID="d59eba41-82be-4721-8082-212b6df76bed" containerID="e7f916a1fbf51f3dba7a8cec0351f2f8d46860f63118cf6b9711eb08d4cc75a4" exitCode=60 Oct 31 19:30:03.413298 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.413231 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerDied","Data":"e7f916a1fbf51f3dba7a8cec0351f2f8d46860f63118cf6b9711eb08d4cc75a4"} Oct 31 19:30:03.413409 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.413385 5688 scope.go:122] "RemoveContainer" containerID="e7f916a1fbf51f3dba7a8cec0351f2f8d46860f63118cf6b9711eb08d4cc75a4" Oct 31 19:30:03.416023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.416003 5688 generic.go:356] "Generic (PLEG): container finished" podID="0980939e-1112-4300-91d8-a4e9437df583" containerID="3d01987b46b29998e3babeb661381e6903cd95cbe7893adbfc83e3c67faa1067" exitCode=60 Oct 31 19:30:03.416099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.416039 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerDied","Data":"3d01987b46b29998e3babeb661381e6903cd95cbe7893adbfc83e3c67faa1067"} Oct 31 19:30:03.416099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.416075 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerStarted","Data":"a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3"} Oct 31 19:30:03.749632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:03.749580 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" podStartSLOduration=3.749564168 podStartE2EDuration="3.749564168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.636821876 +0000 UTC m=+4124.238963020" watchObservedRunningTime="2026-10-31 19:30:03.749564168 +0000 UTC m=+4124.351705312" Oct 31 19:30:04.202295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.202242 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" podStartSLOduration=4.202228969 podStartE2EDuration="4.202228969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:04.201521277 +0000 UTC m=+4124.803662422" watchObservedRunningTime="2026-10-31 19:30:04.202228969 +0000 UTC m=+4124.804370113" Oct 31 19:30:04.354069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.353988 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" containerID="cri-o://56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742" gracePeriod=25 Oct 31 19:30:04.423826 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.423791 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerStarted","Data":"5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9"} Oct 31 19:30:04.425646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.425622 5688 generic.go:356] "Generic (PLEG): container finished" podID="0980939e-1112-4300-91d8-a4e9437df583" containerID="a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3" exitCode=60 Oct 31 19:30:04.425778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.425697 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerDied","Data":"a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3"} Oct 31 19:30:04.425778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.425757 5688 scope.go:122] "RemoveContainer" containerID="3d01987b46b29998e3babeb661381e6903cd95cbe7893adbfc83e3c67faa1067" Oct 31 19:30:04.425920 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.425904 5688 scope.go:122] "RemoveContainer" containerID="a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3" Oct 31 19:30:04.426165 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:04.426138 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz_odh-ai-gateway-infra(0980939e-1112-4300-91d8-a4e9437df583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" podUID="0980939e-1112-4300-91d8-a4e9437df583" Oct 31 19:30:04.427546 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.427527 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerStarted","Data":"e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f"} Oct 31 19:30:04.429380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.429363 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b51bfe4-eaad-401d-924e-636f65470576" containerID="b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64" exitCode=60 Oct 31 19:30:04.429473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.429441 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerDied","Data":"b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64"} Oct 31 19:30:04.429605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.429591 5688 scope.go:122] "RemoveContainer" containerID="b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64" Oct 31 19:30:04.429826 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:04.429798 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs_odh-ai-gateway-infra(2b51bfe4-eaad-401d-924e-636f65470576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" podUID="2b51bfe4-eaad-401d-924e-636f65470576" Oct 31 19:30:04.431224 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.431204 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerStarted","Data":"207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9"} Oct 31 19:30:04.433322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.433272 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerStarted","Data":"a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765"} Oct 31 19:30:04.439467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.439333 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerStarted","Data":"9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8"} Oct 31 19:30:04.499567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.499543 5688 scope.go:122] "RemoveContainer" containerID="fa000aff302e16500b3de676ae84a981e66aa5732b8f6ba968cc98e6080f1055" Oct 31 19:30:04.743765 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.743738 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:30:04.810158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.810130 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume\") pod \"7e95b725-8577-4237-be04-fb88f12b264c\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " Oct 31 19:30:04.810349 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.810254 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh\") pod \"7e95b725-8577-4237-be04-fb88f12b264c\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " Oct 31 19:30:04.810399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.810368 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials\") pod \"7e95b725-8577-4237-be04-fb88f12b264c\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " Oct 31 19:30:04.810466 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.810402 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume\") pod \"7e95b725-8577-4237-be04-fb88f12b264c\" (UID: \"7e95b725-8577-4237-be04-fb88f12b264c\") " Oct 31 19:30:04.810952 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.810908 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume" pod "7e95b725-8577-4237-be04-fb88f12b264c" (UID: "7e95b725-8577-4237-be04-fb88f12b264c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:04.810952 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.810944 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume" pod "7e95b725-8577-4237-be04-fb88f12b264c" (UID: "7e95b725-8577-4237-be04-fb88f12b264c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:04.813072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.813049 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh" pod "7e95b725-8577-4237-be04-fb88f12b264c" (UID: "7e95b725-8577-4237-be04-fb88f12b264c"). InnerVolumeSpecName "kube-api-access-m79qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:04.813233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.813078 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials" pod "7e95b725-8577-4237-be04-fb88f12b264c" (UID: "7e95b725-8577-4237-be04-fb88f12b264c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:04.912278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.912249 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:04.912278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.912275 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m79qh\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-kube-api-access-m79qh\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:04.912451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.912289 5688 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e95b725-8577-4237-be04-fb88f12b264c-provider-credentials\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:04.912451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:04.912299 5688 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e95b725-8577-4237-be04-fb88f12b264c-routing-overlay-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:05.451229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.451163 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerID="a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765" exitCode=60 Oct 31 19:30:05.451724 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.451251 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerDied","Data":"a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765"} Oct 31 19:30:05.451724 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.451328 5688 scope.go:122] "RemoveContainer" containerID="f8bb45cecc1b10e8c9a8087a589f755a55c8ed4b8b27690a2ad77cd670ab934c" Oct 31 19:30:05.451724 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.451558 5688 scope.go:122] "RemoveContainer" containerID="a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765" Oct 31 19:30:05.451867 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.451845 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891250-f8cxd_odh-ai-gateway-infra(2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" Oct 31 19:30:05.453572 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.453550 5688 generic.go:356] "Generic (PLEG): container finished" podID="87498025-815e-4cec-b6db-a79423447931" containerID="9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8" exitCode=60 Oct 31 19:30:05.453690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.453582 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerDied","Data":"9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8"} Oct 31 19:30:05.453845 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.453830 5688 scope.go:122] "RemoveContainer" containerID="9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8" Oct 31 19:30:05.454108 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.454081 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8_odh-ai-gateway-infra(87498025-815e-4cec-b6db-a79423447931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" podUID="87498025-815e-4cec-b6db-a79423447931" Oct 31 19:30:05.455251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.455231 5688 generic.go:356] "Generic (PLEG): container finished" podID="d59eba41-82be-4721-8082-212b6df76bed" containerID="5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9" exitCode=60 Oct 31 19:30:05.455357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.455311 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerDied","Data":"5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9"} Oct 31 19:30:05.455470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.455456 5688 scope.go:122] "RemoveContainer" containerID="5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9" Oct 31 19:30:05.455680 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.455661 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx_odh-ai-gateway-infra(d59eba41-82be-4721-8082-212b6df76bed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" podUID="d59eba41-82be-4721-8082-212b6df76bed" Oct 31 19:30:05.456627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.456606 5688 generic.go:356] "Generic (PLEG): container finished" podID="7e95b725-8577-4237-be04-fb88f12b264c" containerID="56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742" exitCode=0 Oct 31 19:30:05.456759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.456648 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" Oct 31 19:30:05.456759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.456685 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" event={"ID":"7e95b725-8577-4237-be04-fb88f12b264c","Type":"ContainerDied","Data":"56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742"} Oct 31 19:30:05.456759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.456730 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr" event={"ID":"7e95b725-8577-4237-be04-fb88f12b264c","Type":"ContainerDied","Data":"c644148fd19663a27ace4819b14ea2597e764d631d275258ebfc3806140448d7"} Oct 31 19:30:05.458680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.458661 5688 scope.go:122] "RemoveContainer" containerID="a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3" Oct 31 19:30:05.458877 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.458844 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz_odh-ai-gateway-infra(0980939e-1112-4300-91d8-a4e9437df583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" podUID="0980939e-1112-4300-91d8-a4e9437df583" Oct 31 19:30:05.460374 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.460349 5688 generic.go:356] "Generic (PLEG): container finished" podID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerID="e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f" exitCode=60 Oct 31 19:30:05.460496 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.460437 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerDied","Data":"e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f"} Oct 31 19:30:05.460596 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.460578 5688 scope.go:122] "RemoveContainer" containerID="e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f" Oct 31 19:30:05.460818 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.460780 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc_odh-ai-gateway-infra(f6c7051a-1c9b-457b-9bde-b83e3878f2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" Oct 31 19:30:05.462436 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.462416 5688 scope.go:122] "RemoveContainer" containerID="b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64" Oct 31 19:30:05.462660 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.462637 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs_odh-ai-gateway-infra(2b51bfe4-eaad-401d-924e-636f65470576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" podUID="2b51bfe4-eaad-401d-924e-636f65470576" Oct 31 19:30:05.464268 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.464242 5688 generic.go:356] "Generic (PLEG): container finished" podID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerID="207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9" exitCode=60 Oct 31 19:30:05.464383 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.464276 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerDied","Data":"207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9"} Oct 31 19:30:05.464519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.464503 5688 scope.go:122] "RemoveContainer" containerID="207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9" Oct 31 19:30:05.464720 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.464702 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr_odh-ai-gateway-infra(6e2f39c2-09dc-4648-803d-9fd91d3b091d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" Oct 31 19:30:05.467731 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.467715 5688 scope.go:122] "RemoveContainer" containerID="aa312709201cbb5e8c7a9391a3f6902fa1e5b699a9ddcc847ebe37e4bbb05735" Oct 31 19:30:05.479996 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.479980 5688 scope.go:122] "RemoveContainer" containerID="e7f916a1fbf51f3dba7a8cec0351f2f8d46860f63118cf6b9711eb08d4cc75a4" Oct 31 19:30:05.493299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.493275 5688 scope.go:122] "RemoveContainer" containerID="56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742" Oct 31 19:30:05.501844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.501823 5688 scope.go:122] "RemoveContainer" containerID="56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742" Oct 31 19:30:05.502108 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:05.502089 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742\": container with ID starting with 56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742 not found: ID does not exist" containerID="56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742" Oct 31 19:30:05.502218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.502118 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742"} err="failed to get container status \"56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742\": rpc error: code = NotFound desc = could not find container \"56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742\": container with ID starting with 56fc6302ad5ea3e7826bcef4d93c29878adab5f202f84d77dd674c53dc81f742 not found: ID does not exist" Oct 31 19:30:05.502218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.502145 5688 scope.go:122] "RemoveContainer" containerID="c3cb81e7e522bdcc4cf188ba42220d2f8dde3f76dd151c61e0db38ae294c8dc0" Oct 31 19:30:05.514017 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.514000 5688 scope.go:122] "RemoveContainer" containerID="d560146bfa9bcdcede355b9c8fad49f4e2f28a55b51efb71821fee782beff98e" Oct 31 19:30:05.676637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.676601 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr"] Oct 31 19:30:05.676804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:05.676657 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c897f8c6f-lsxmr"] Oct 31 19:30:06.474399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:06.474375 5688 scope.go:122] "RemoveContainer" containerID="5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9" Oct 31 19:30:06.474824 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:06.474627 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx_odh-ai-gateway-infra(d59eba41-82be-4721-8082-212b6df76bed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" podUID="d59eba41-82be-4721-8082-212b6df76bed" Oct 31 19:30:06.477556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:06.477527 5688 scope.go:122] "RemoveContainer" containerID="e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f" Oct 31 19:30:06.477780 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:06.477755 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc_odh-ai-gateway-infra(f6c7051a-1c9b-457b-9bde-b83e3878f2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" Oct 31 19:30:06.478954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:06.478939 5688 scope.go:122] "RemoveContainer" containerID="207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9" Oct 31 19:30:06.479144 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:06.479121 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr_odh-ai-gateway-infra(6e2f39c2-09dc-4648-803d-9fd91d3b091d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" Oct 31 19:30:06.480445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:06.480429 5688 scope.go:122] "RemoveContainer" containerID="a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765" Oct 31 19:30:06.480587 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:06.480570 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891250-f8cxd_odh-ai-gateway-infra(2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" Oct 31 19:30:06.481908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:06.481892 5688 scope.go:122] "RemoveContainer" containerID="9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8" Oct 31 19:30:06.482054 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:06.482038 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8_odh-ai-gateway-infra(87498025-815e-4cec-b6db-a79423447931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" podUID="87498025-815e-4cec-b6db-a79423447931" Oct 31 19:30:07.567789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:07.567756 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e95b725-8577-4237-be04-fb88f12b264c" path="/var/lib/kubelet/pods/7e95b725-8577-4237-be04-fb88f12b264c/volumes" Oct 31 19:30:14.559738 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:14.559708 5688 scope.go:122] "RemoveContainer" containerID="a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3" Oct 31 19:30:15.537927 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:15.537897 5688 generic.go:356] "Generic (PLEG): container finished" podID="0980939e-1112-4300-91d8-a4e9437df583" containerID="7477e6af84d833dfb86f411de1ebf7021231638f843f066f0ded9e503735ed79" exitCode=60 Oct 31 19:30:15.538080 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:15.537955 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerDied","Data":"7477e6af84d833dfb86f411de1ebf7021231638f843f066f0ded9e503735ed79"} Oct 31 19:30:15.538080 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:15.538001 5688 scope.go:122] "RemoveContainer" containerID="a5f5bd11910bf20bdefb12430eef85e507188896d976718cae4bbc079a731dd3" Oct 31 19:30:15.538400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:15.538380 5688 scope.go:122] "RemoveContainer" containerID="7477e6af84d833dfb86f411de1ebf7021231638f843f066f0ded9e503735ed79" Oct 31 19:30:15.538676 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:15.538656 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz_odh-ai-gateway-infra(0980939e-1112-4300-91d8-a4e9437df583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" podUID="0980939e-1112-4300-91d8-a4e9437df583" Oct 31 19:30:16.559935 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:16.559901 5688 scope.go:122] "RemoveContainer" containerID="5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9" Oct 31 19:30:16.560450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:16.559943 5688 scope.go:122] "RemoveContainer" containerID="207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9" Oct 31 19:30:16.560450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:16.560121 5688 scope.go:122] "RemoveContainer" containerID="9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8" Oct 31 19:30:16.560450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:16.560257 5688 scope.go:122] "RemoveContainer" containerID="e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f" Oct 31 19:30:16.660833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:16.660810 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz"] Oct 31 19:30:16.880105 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:16.880082 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:17.063410 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.063384 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6\" (UniqueName: \"kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6\") pod \"0980939e-1112-4300-91d8-a4e9437df583\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " Oct 31 19:30:17.063527 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.063477 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca\") pod \"0980939e-1112-4300-91d8-a4e9437df583\" (UID: \"0980939e-1112-4300-91d8-a4e9437df583\") " Oct 31 19:30:17.063810 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.063788 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca" pod "0980939e-1112-4300-91d8-a4e9437df583" (UID: "0980939e-1112-4300-91d8-a4e9437df583"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:17.063972 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.063956 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0980939e-1112-4300-91d8-a4e9437df583-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:17.065394 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.065375 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6" pod "0980939e-1112-4300-91d8-a4e9437df583" (UID: "0980939e-1112-4300-91d8-a4e9437df583"). InnerVolumeSpecName "kube-api-access-dwdz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:17.165004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.164936 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwdz6\" (UniqueName: \"kubernetes.io/projected/0980939e-1112-4300-91d8-a4e9437df583-kube-api-access-dwdz6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:17.552810 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.552785 5688 generic.go:356] "Generic (PLEG): container finished" podID="d59eba41-82be-4721-8082-212b6df76bed" containerID="57b2e6bd61d9ca4013371f86e4a272b5e0811deb03d82776647988d9498a9fa3" exitCode=60 Oct 31 19:30:17.552917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.552853 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerDied","Data":"57b2e6bd61d9ca4013371f86e4a272b5e0811deb03d82776647988d9498a9fa3"} Oct 31 19:30:17.552917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.552910 5688 scope.go:122] "RemoveContainer" containerID="5044ae7e8816d4c3446d1fceade5c98bf4e54696849af8147c3cfd7f2e5ff4e9" Oct 31 19:30:17.553317 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.553300 5688 scope.go:122] "RemoveContainer" containerID="57b2e6bd61d9ca4013371f86e4a272b5e0811deb03d82776647988d9498a9fa3" Oct 31 19:30:17.553597 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:17.553576 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx_odh-ai-gateway-infra(d59eba41-82be-4721-8082-212b6df76bed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" podUID="d59eba41-82be-4721-8082-212b6df76bed" Oct 31 19:30:17.554569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.554550 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" event={"ID":"0980939e-1112-4300-91d8-a4e9437df583","Type":"ContainerDied","Data":"e6ed873718dfd18964fa2ac268a15e24d8fe53309b6901f0accd3c69d4c9b223"} Oct 31 19:30:17.554671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.554555 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:17.556484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.556465 5688 generic.go:356] "Generic (PLEG): container finished" podID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerID="36c9835f41fab31974afaa23d77147862bb239ded0266e8d63704dcdfbfc77ce" exitCode=60 Oct 31 19:30:17.556584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.556519 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerDied","Data":"36c9835f41fab31974afaa23d77147862bb239ded0266e8d63704dcdfbfc77ce"} Oct 31 19:30:17.556897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.556873 5688 scope.go:122] "RemoveContainer" containerID="36c9835f41fab31974afaa23d77147862bb239ded0266e8d63704dcdfbfc77ce" Oct 31 19:30:17.557119 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:17.557099 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc_odh-ai-gateway-infra(f6c7051a-1c9b-457b-9bde-b83e3878f2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" Oct 31 19:30:17.559316 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.559295 5688 generic.go:356] "Generic (PLEG): container finished" podID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerID="6f9bf4329117bb5f6b090bd3e5aa9131e310c6cdd7c28326acfa26b52ab4abb8" exitCode=60 Oct 31 19:30:17.559766 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.559750 5688 scope.go:122] "RemoveContainer" containerID="b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64" Oct 31 19:30:17.562637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.562620 5688 generic.go:356] "Generic (PLEG): container finished" podID="87498025-815e-4cec-b6db-a79423447931" containerID="bc8f56c1863433618a5722cb74e6ba87f87b310e8e8273ea91b9ad8061ae3533" exitCode=60 Oct 31 19:30:17.569530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.569492 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerDied","Data":"6f9bf4329117bb5f6b090bd3e5aa9131e310c6cdd7c28326acfa26b52ab4abb8"} Oct 31 19:30:17.569645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.569562 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerDied","Data":"bc8f56c1863433618a5722cb74e6ba87f87b310e8e8273ea91b9ad8061ae3533"} Oct 31 19:30:17.569883 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.569865 5688 scope.go:122] "RemoveContainer" containerID="bc8f56c1863433618a5722cb74e6ba87f87b310e8e8273ea91b9ad8061ae3533" Oct 31 19:30:17.569970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.569955 5688 scope.go:122] "RemoveContainer" containerID="6f9bf4329117bb5f6b090bd3e5aa9131e310c6cdd7c28326acfa26b52ab4abb8" Oct 31 19:30:17.570088 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:17.570071 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8_odh-ai-gateway-infra(87498025-815e-4cec-b6db-a79423447931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" podUID="87498025-815e-4cec-b6db-a79423447931" Oct 31 19:30:17.570163 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:17.570145 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr_odh-ai-gateway-infra(6e2f39c2-09dc-4648-803d-9fd91d3b091d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" Oct 31 19:30:17.677820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.677799 5688 scope.go:122] "RemoveContainer" containerID="7477e6af84d833dfb86f411de1ebf7021231638f843f066f0ded9e503735ed79" Oct 31 19:30:17.693649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.693536 5688 scope.go:122] "RemoveContainer" containerID="e35150d3594e72cd647920c9180c682bf26f110171455825ed0c0a4d936fde0f" Oct 31 19:30:17.706538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.706524 5688 scope.go:122] "RemoveContainer" containerID="207623c54817f97f6cc516bf117c075593f9bc4ee27b64ff51c96b7d1ad7dfb9" Oct 31 19:30:17.718278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:17.718256 5688 scope.go:122] "RemoveContainer" containerID="9f740935894af861a0f4693ef9e7d20cd9f2d467e86aeded0f4eb91ce72c08d8" Oct 31 19:30:18.490469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.490435 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8"] Oct 31 19:30:18.566251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.565885 5688 scope.go:122] "RemoveContainer" containerID="a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765" Oct 31 19:30:18.583097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.583065 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerStarted","Data":"e1ce4867096392f22b956b42efb6f974dc2f37eb881831e0b26f3416424d14b7"} Oct 31 19:30:18.722223 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.711501 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx"] Oct 31 19:30:18.851073 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.851039 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:18.889953 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.889920 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca\") pod \"87498025-815e-4cec-b6db-a79423447931\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " Oct 31 19:30:18.890147 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.890091 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t\" (UniqueName: \"kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t\") pod \"87498025-815e-4cec-b6db-a79423447931\" (UID: \"87498025-815e-4cec-b6db-a79423447931\") " Oct 31 19:30:18.891135 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.891017 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca" pod "87498025-815e-4cec-b6db-a79423447931" (UID: "87498025-815e-4cec-b6db-a79423447931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:18.897746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.897703 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t" pod "87498025-815e-4cec-b6db-a79423447931" (UID: "87498025-815e-4cec-b6db-a79423447931"). InnerVolumeSpecName "kube-api-access-7fv6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:18.975931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.975899 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr"] Oct 31 19:30:18.981993 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.981970 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:18.991014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.990992 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca\") pod \"d59eba41-82be-4721-8082-212b6df76bed\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " Oct 31 19:30:18.991148 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.991131 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8\" (UniqueName: \"kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8\") pod \"d59eba41-82be-4721-8082-212b6df76bed\" (UID: \"d59eba41-82be-4721-8082-212b6df76bed\") " Oct 31 19:30:18.991365 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.991341 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca" pod "d59eba41-82be-4721-8082-212b6df76bed" (UID: "d59eba41-82be-4721-8082-212b6df76bed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:18.991487 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.991472 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59eba41-82be-4721-8082-212b6df76bed-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:18.991556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.991490 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87498025-815e-4cec-b6db-a79423447931-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:18.991556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.991499 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fv6t\" (UniqueName: \"kubernetes.io/projected/87498025-815e-4cec-b6db-a79423447931-kube-api-access-7fv6t\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:18.993602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:18.993579 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8" pod "d59eba41-82be-4721-8082-212b6df76bed" (UID: "d59eba41-82be-4721-8082-212b6df76bed"). InnerVolumeSpecName "kube-api-access-lbrv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:19.092513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.092466 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbrv8\" (UniqueName: \"kubernetes.io/projected/d59eba41-82be-4721-8082-212b6df76bed-kube-api-access-lbrv8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:19.098251 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.097150 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc"] Oct 31 19:30:19.118028 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.117775 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:19.198698 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.198342 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw\" (UniqueName: \"kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw\") pod \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " Oct 31 19:30:19.198698 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.198487 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca\") pod \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\" (UID: \"6e2f39c2-09dc-4648-803d-9fd91d3b091d\") " Oct 31 19:30:19.199272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.199235 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca" pod "6e2f39c2-09dc-4648-803d-9fd91d3b091d" (UID: "6e2f39c2-09dc-4648-803d-9fd91d3b091d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:19.201356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.201334 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw" pod "6e2f39c2-09dc-4648-803d-9fd91d3b091d" (UID: "6e2f39c2-09dc-4648-803d-9fd91d3b091d"). InnerVolumeSpecName "kube-api-access-zxbqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:19.288021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.268748 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:19.322207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.299585 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca\") pod \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " Oct 31 19:30:19.322207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.299698 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6\" (UniqueName: \"kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6\") pod \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\" (UID: \"f6c7051a-1c9b-457b-9bde-b83e3878f2c2\") " Oct 31 19:30:19.322207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.300086 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbqw\" (UniqueName: \"kubernetes.io/projected/6e2f39c2-09dc-4648-803d-9fd91d3b091d-kube-api-access-zxbqw\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:19.322207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.300104 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f39c2-09dc-4648-803d-9fd91d3b091d-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:19.322207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.300687 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca" pod "f6c7051a-1c9b-457b-9bde-b83e3878f2c2" (UID: "f6c7051a-1c9b-457b-9bde-b83e3878f2c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:19.329015 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.322759 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6" pod "f6c7051a-1c9b-457b-9bde-b83e3878f2c2" (UID: "f6c7051a-1c9b-457b-9bde-b83e3878f2c2"). InnerVolumeSpecName "kube-api-access-pvrn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:19.400816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.400787 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:19.400816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.400813 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrn6\" (UniqueName: \"kubernetes.io/projected/f6c7051a-1c9b-457b-9bde-b83e3878f2c2-kube-api-access-pvrn6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:19.597690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.597667 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" Oct 31 19:30:19.598050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.597698 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx" event={"ID":"d59eba41-82be-4721-8082-212b6df76bed","Type":"ContainerDied","Data":"4d85d61e54206c6e77f5153b1d7ab5453a3e1731840c16427f553f0e232596f7"} Oct 31 19:30:19.598050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.597760 5688 scope.go:122] "RemoveContainer" containerID="57b2e6bd61d9ca4013371f86e4a272b5e0811deb03d82776647988d9498a9fa3" Oct 31 19:30:19.600043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.599573 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" Oct 31 19:30:19.600043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.599696 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc" event={"ID":"f6c7051a-1c9b-457b-9bde-b83e3878f2c2","Type":"ContainerDied","Data":"226d061a527edd90dec5d8f19b80493526471f0cbaf72fa61217a9e06fa33f30"} Oct 31 19:30:19.604542 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.604522 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b51bfe4-eaad-401d-924e-636f65470576" containerID="e1ce4867096392f22b956b42efb6f974dc2f37eb881831e0b26f3416424d14b7" exitCode=60 Oct 31 19:30:19.604667 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.604606 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerDied","Data":"e1ce4867096392f22b956b42efb6f974dc2f37eb881831e0b26f3416424d14b7"} Oct 31 19:30:19.604945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.604927 5688 scope.go:122] "RemoveContainer" containerID="e1ce4867096392f22b956b42efb6f974dc2f37eb881831e0b26f3416424d14b7" Oct 31 19:30:19.605161 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:19.605135 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs_odh-ai-gateway-infra(2b51bfe4-eaad-401d-924e-636f65470576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" podUID="2b51bfe4-eaad-401d-924e-636f65470576" Oct 31 19:30:19.605908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.605893 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" Oct 31 19:30:19.606004 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.605934 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr" event={"ID":"6e2f39c2-09dc-4648-803d-9fd91d3b091d","Type":"ContainerDied","Data":"38eb24b18d0d6356084496e9c90daa0a19d50a56e74e191f977d8fe7bf378c19"} Oct 31 19:30:19.608031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.608013 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerID="56f83cdc74f363c1053d5c2d109975046dda3645c575ddce9590813b56de9659" exitCode=60 Oct 31 19:30:19.608145 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.608083 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerDied","Data":"56f83cdc74f363c1053d5c2d109975046dda3645c575ddce9590813b56de9659"} Oct 31 19:30:19.608438 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.608422 5688 scope.go:122] "RemoveContainer" containerID="56f83cdc74f363c1053d5c2d109975046dda3645c575ddce9590813b56de9659" Oct 31 19:30:19.608678 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:19.608592 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891250-f8cxd_odh-ai-gateway-infra(2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" Oct 31 19:30:19.608732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.608673 5688 scope.go:122] "RemoveContainer" containerID="36c9835f41fab31974afaa23d77147862bb239ded0266e8d63704dcdfbfc77ce" Oct 31 19:30:19.609333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.609314 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" event={"ID":"87498025-815e-4cec-b6db-a79423447931","Type":"ContainerDied","Data":"a5b8d04eceefbaac0de7e0873942fc1cc720caabf49907dd233d8b4f9db03084"} Oct 31 19:30:19.609418 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.609336 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8" Oct 31 19:30:19.617245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.617229 5688 scope.go:122] "RemoveContainer" containerID="b197524cb683a66e40252e55df9b09503263fb6e24a32748c957b38127655a64" Oct 31 19:30:19.629062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.629048 5688 scope.go:122] "RemoveContainer" containerID="6f9bf4329117bb5f6b090bd3e5aa9131e310c6cdd7c28326acfa26b52ab4abb8" Oct 31 19:30:19.637025 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.637008 5688 scope.go:122] "RemoveContainer" containerID="a1418e0e6cd51858b2f879203b49cd2983b33554fed89ceb8c8da3f8b937c765" Oct 31 19:30:19.655730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.655714 5688 scope.go:122] "RemoveContainer" containerID="bc8f56c1863433618a5722cb74e6ba87f87b310e8e8273ea91b9ad8061ae3533" Oct 31 19:30:19.716146 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.716118 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs"] Oct 31 19:30:19.732994 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.732969 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8"] Oct 31 19:30:19.742114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.742087 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b70248-29891250-fsxv8"] Oct 31 19:30:19.852520 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.852458 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx"] Oct 31 19:30:19.857830 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.857807 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891250-4qgrx"] Oct 31 19:30:19.968741 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.968712 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr"] Oct 31 19:30:19.973222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:19.973167 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fdd317-29891250-hp8zr"] Oct 31 19:30:20.320747 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.320716 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc"] Oct 31 19:30:20.322886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.322865 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1aefcd-29891250-mbrwc"] Oct 31 19:30:20.785725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.785702 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:20.815262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.815237 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm\" (UniqueName: \"kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm\") pod \"2b51bfe4-eaad-401d-924e-636f65470576\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " Oct 31 19:30:20.815389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.815367 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca\") pod \"2b51bfe4-eaad-401d-924e-636f65470576\" (UID: \"2b51bfe4-eaad-401d-924e-636f65470576\") " Oct 31 19:30:20.815683 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.815662 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca" pod "2b51bfe4-eaad-401d-924e-636f65470576" (UID: "2b51bfe4-eaad-401d-924e-636f65470576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:20.815897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.815859 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b51bfe4-eaad-401d-924e-636f65470576-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:20.817455 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.817433 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm" pod "2b51bfe4-eaad-401d-924e-636f65470576" (UID: "2b51bfe4-eaad-401d-924e-636f65470576"). InnerVolumeSpecName "kube-api-access-xxwsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:20.916985 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:20.916905 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwsm\" (UniqueName: \"kubernetes.io/projected/2b51bfe4-eaad-401d-924e-636f65470576-kube-api-access-xxwsm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:21.096529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.096501 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd"] Oct 31 19:30:21.258676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.258650 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:21.320472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.320437 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca\") pod \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " Oct 31 19:30:21.320615 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.320595 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg\" (UniqueName: \"kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg\") pod \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\" (UID: \"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1\") " Oct 31 19:30:21.320776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.320746 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca" pod "2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" (UID: "2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:21.321094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.321079 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:21.322828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.322806 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg" pod "2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" (UID: "2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1"). InnerVolumeSpecName "kube-api-access-g56fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:21.421876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.421844 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56fg\" (UniqueName: \"kubernetes.io/projected/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1-kube-api-access-g56fg\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:21.566915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.566841 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" path="/var/lib/kubelet/pods/6e2f39c2-09dc-4648-803d-9fd91d3b091d/volumes" Oct 31 19:30:21.567249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.567235 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87498025-815e-4cec-b6db-a79423447931" path="/var/lib/kubelet/pods/87498025-815e-4cec-b6db-a79423447931/volumes" Oct 31 19:30:21.567624 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.567609 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59eba41-82be-4721-8082-212b6df76bed" path="/var/lib/kubelet/pods/d59eba41-82be-4721-8082-212b6df76bed/volumes" Oct 31 19:30:21.568261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.568246 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" path="/var/lib/kubelet/pods/f6c7051a-1c9b-457b-9bde-b83e3878f2c2/volumes" Oct 31 19:30:21.629921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.629890 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" event={"ID":"2b51bfe4-eaad-401d-924e-636f65470576","Type":"ContainerDied","Data":"f455d384e1b33d1ed83e643ca9247969dbb75cc2984bc049d768a86c2ed39c25"} Oct 31 19:30:21.630046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.629899 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs" Oct 31 19:30:21.630046 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.629957 5688 scope.go:122] "RemoveContainer" containerID="e1ce4867096392f22b956b42efb6f974dc2f37eb881831e0b26f3416424d14b7" Oct 31 19:30:21.632231 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.632028 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" event={"ID":"2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1","Type":"ContainerDied","Data":"ec58451537d259d5a6eebb8a4d1ae1821d73219a10f7c9f18454312f80771f8b"} Oct 31 19:30:21.632231 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.632067 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd" Oct 31 19:30:21.639627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.639611 5688 scope.go:122] "RemoveContainer" containerID="56f83cdc74f363c1053d5c2d109975046dda3645c575ddce9590813b56de9659" Oct 31 19:30:21.793112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.793082 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd"] Oct 31 19:30:21.799247 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.799223 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-f8cxd"] Oct 31 19:30:21.907157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.907090 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs"] Oct 31 19:30:21.915326 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:21.915299 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d1ff32-29891250-55vhs"] Oct 31 19:30:22.766205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:22.766153 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8"] Oct 31 19:30:22.766445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:22.766415 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" containerID="cri-o://1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b" gracePeriod=30 Oct 31 19:30:23.566074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:23.566033 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" path="/var/lib/kubelet/pods/2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1/volumes" Oct 31 19:30:23.566508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:23.566491 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b51bfe4-eaad-401d-924e-636f65470576" path="/var/lib/kubelet/pods/2b51bfe4-eaad-401d-924e-636f65470576/volumes" Oct 31 19:30:24.904816 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:24.904783 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:30:28.035486 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.035430 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:30:28.092719 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092694 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.092868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092731 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.092868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092782 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.092868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092828 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.093012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092880 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.093012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092924 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.093012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092941 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:30:28.093012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.092963 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.093261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093050 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:30:28.093261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093063 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.093261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093101 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf\") pod \"7f134f9d-1862-48db-95d4-843d1a967dd9\" (UID: \"7f134f9d-1862-48db-95d4-843d1a967dd9\") " Oct 31 19:30:28.093261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093217 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:30:28.093467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093287 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:28.093586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093538 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f134f9d-1862-48db-95d4-843d1a967dd9-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.093586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093559 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.093586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093575 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.093586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093588 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.093909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.093886 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:30:28.095273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.095245 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:30:28.095384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.095322 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "kube-api-access-l4rwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:28.095384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.095336 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:30:28.095384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.095376 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token" pod "7f134f9d-1862-48db-95d4-843d1a967dd9" (UID: "7f134f9d-1862-48db-95d4-843d1a967dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:28.194101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.194079 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.194101 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.194100 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.194250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.194108 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.194250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.194119 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f134f9d-1862-48db-95d4-843d1a967dd9-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.194250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.194127 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rwf\" (UniqueName: \"kubernetes.io/projected/7f134f9d-1862-48db-95d4-843d1a967dd9-kube-api-access-l4rwf\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:28.669453 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.669420 5688 generic.go:356] "Generic (PLEG): container finished" podID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerID="1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b" exitCode=0 Oct 31 19:30:28.669625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.669463 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" Oct 31 19:30:28.669625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.669506 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" event={"ID":"7f134f9d-1862-48db-95d4-843d1a967dd9","Type":"ContainerDied","Data":"1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b"} Oct 31 19:30:28.669625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.669561 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8" event={"ID":"7f134f9d-1862-48db-95d4-843d1a967dd9","Type":"ContainerDied","Data":"8e380952d1cb7ca33d82ae84bdc375c43953cac06ff45cf55320319be818c800"} Oct 31 19:30:28.669625 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.669583 5688 scope.go:122] "RemoveContainer" containerID="1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b" Oct 31 19:30:28.680123 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.680106 5688 scope.go:122] "RemoveContainer" containerID="c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220" Oct 31 19:30:28.689059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.689042 5688 scope.go:122] "RemoveContainer" containerID="1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b" Oct 31 19:30:28.689349 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:28.689327 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b\": container with ID starting with 1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b not found: ID does not exist" containerID="1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b" Oct 31 19:30:28.689457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.689358 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b"} err="failed to get container status \"1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b\": rpc error: code = NotFound desc = could not find container \"1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b\": container with ID starting with 1b473e15c50ff7541051602b09b1577f3ffb1e5a055822bc4431cafe9024910b not found: ID does not exist" Oct 31 19:30:28.689457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.689376 5688 scope.go:122] "RemoveContainer" containerID="c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220" Oct 31 19:30:28.689651 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:28.689629 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220\": container with ID starting with c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220 not found: ID does not exist" containerID="c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220" Oct 31 19:30:28.689713 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.689656 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220"} err="failed to get container status \"c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220\": rpc error: code = NotFound desc = could not find container \"c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220\": container with ID starting with c7f044c12ee725cfbe2b5dd5ea9232a11361bb3edc93d8c17b5857be959ca220 not found: ID does not exist" Oct 31 19:30:28.883194 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.883146 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8"] Oct 31 19:30:28.932519 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:28.932452 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b70248-openshift-default-6ccf86765-5kxx8"] Oct 31 19:30:29.567478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:29.567449 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" path="/var/lib/kubelet/pods/7f134f9d-1862-48db-95d4-843d1a967dd9/volumes" Oct 31 19:30:34.563209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:34.561406 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf"] Oct 31 19:30:34.563209 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:34.562390 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" podUID="d8c6be1b-4020-4e57-97b7-95efa57d12fc" containerName="maas-api" containerID="cri-o://8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3" gracePeriod=30 Oct 31 19:30:35.522405 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.522368 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:30:35.635647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.634914 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca\") pod \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " Oct 31 19:30:35.635647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.635028 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b\" (UniqueName: \"kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b\") pod \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " Oct 31 19:30:35.635647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.635071 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\") pod \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " Oct 31 19:30:35.635647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.635226 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs\") pod \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\" (UID: \"d8c6be1b-4020-4e57-97b7-95efa57d12fc\") " Oct 31 19:30:35.635647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.635288 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca" pod "d8c6be1b-4020-4e57-97b7-95efa57d12fc" (UID: "d8c6be1b-4020-4e57-97b7-95efa57d12fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:35.635647 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.635611 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6be1b-4020-4e57-97b7-95efa57d12fc-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:35.667431 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.659268 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs" pod "d8c6be1b-4020-4e57-97b7-95efa57d12fc" (UID: "d8c6be1b-4020-4e57-97b7-95efa57d12fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:30:35.681206 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.678675 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b" pod "d8c6be1b-4020-4e57-97b7-95efa57d12fc" (UID: "d8c6be1b-4020-4e57-97b7-95efa57d12fc"). InnerVolumeSpecName "kube-api-access-vpp5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:35.685245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.684448 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls" pod "d8c6be1b-4020-4e57-97b7-95efa57d12fc" (UID: "d8c6be1b-4020-4e57-97b7-95efa57d12fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:30:35.789835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.789702 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpp5b\" (UniqueName: \"kubernetes.io/projected/d8c6be1b-4020-4e57-97b7-95efa57d12fc-kube-api-access-vpp5b\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:35.789835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.789840 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:35.790099 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.789857 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6be1b-4020-4e57-97b7-95efa57d12fc-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:30:35.869219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.868841 5688 generic.go:356] "Generic (PLEG): container finished" podID="d8c6be1b-4020-4e57-97b7-95efa57d12fc" containerID="8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3" exitCode=0 Oct 31 19:30:35.869219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.868916 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" event={"ID":"d8c6be1b-4020-4e57-97b7-95efa57d12fc","Type":"ContainerDied","Data":"8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3"} Oct 31 19:30:35.869219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.868967 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" event={"ID":"d8c6be1b-4020-4e57-97b7-95efa57d12fc","Type":"ContainerDied","Data":"2bb79d57d41dbf34f8e5749c0c83ac6725b9db9fde58d8fcac83ea34fde75d90"} Oct 31 19:30:35.869219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.868998 5688 scope.go:122] "RemoveContainer" containerID="8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3" Oct 31 19:30:35.869610 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.869446 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf" Oct 31 19:30:35.943415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.932571 5688 scope.go:122] "RemoveContainer" containerID="8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3" Oct 31 19:30:35.943415 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:35.943295 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3\": container with ID starting with 8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3 not found: ID does not exist" containerID="8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3" Oct 31 19:30:35.943415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:35.943330 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3"} err="failed to get container status \"8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3\": rpc error: code = NotFound desc = could not find container \"8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3\": container with ID starting with 8a6197ce1a62a68665fb3ee97d621897fd8c7d74c7d13b1752f90d089de981d3 not found: ID does not exist" Oct 31 19:30:36.046410 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:36.046372 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf"] Oct 31 19:30:36.097123 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:36.097086 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c249cd-845d5cfb85-8shlf"] Oct 31 19:30:36.324313 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:36.324282 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:30:36.324495 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:36.324345 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:30:37.566444 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:37.566413 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c6be1b-4020-4e57-97b7-95efa57d12fc" path="/var/lib/kubelet/pods/d8c6be1b-4020-4e57-97b7-95efa57d12fc/volumes" Oct 31 19:30:47.673494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:47.673460 5688 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod0980939e-1112-4300-91d8-a4e9437df583"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod0980939e-1112-4300-91d8-a4e9437df583] : Timed out while waiting for systemd to remove kubepods-burstable-pod0980939e_1112_4300_91d8_a4e9437df583.slice" Oct 31 19:30:47.673955 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:47.673497 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod0980939e-1112-4300-91d8-a4e9437df583] : unable to destroy cgroup paths for cgroup [kubepods burstable pod0980939e-1112-4300-91d8-a4e9437df583] : Timed out while waiting for systemd to remove kubepods-burstable-pod0980939e_1112_4300_91d8_a4e9437df583.slice" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" podUID="0980939e-1112-4300-91d8-a4e9437df583" Oct 31 19:30:47.934673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:47.934599 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz" Oct 31 19:30:48.155306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:48.155270 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz"] Oct 31 19:30:48.160423 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:48.160394 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891250-25ngz"] Oct 31 19:30:49.566129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:30:49.566100 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0980939e-1112-4300-91d8-a4e9437df583" path="/var/lib/kubelet/pods/0980939e-1112-4300-91d8-a4e9437df583/volumes" Oct 31 19:30:54.903130 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:30:54.903093 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:31:08.097306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:08.097271 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm"] Oct 31 19:31:13.098447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.098389 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" podUID="a178472e-7915-49a6-b62e-bed8d86e3a54" containerName="payload-pre-processing" containerID="cri-o://010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b" gracePeriod=25 Oct 31 19:31:13.366798 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.366776 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:31:13.504074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.504048 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25\" (UniqueName: \"kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25\") pod \"a178472e-7915-49a6-b62e-bed8d86e3a54\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " Oct 31 19:31:13.504272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.504132 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume\") pod \"a178472e-7915-49a6-b62e-bed8d86e3a54\" (UID: \"a178472e-7915-49a6-b62e-bed8d86e3a54\") " Oct 31 19:31:13.504501 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.504482 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume" pod "a178472e-7915-49a6-b62e-bed8d86e3a54" (UID: "a178472e-7915-49a6-b62e-bed8d86e3a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:13.506373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.506346 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25" pod "a178472e-7915-49a6-b62e-bed8d86e3a54" (UID: "a178472e-7915-49a6-b62e-bed8d86e3a54"). InnerVolumeSpecName "kube-api-access-chh25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:13.605891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.605828 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chh25\" (UniqueName: \"kubernetes.io/projected/a178472e-7915-49a6-b62e-bed8d86e3a54-kube-api-access-chh25\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:31:13.605891 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:13.605855 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a178472e-7915-49a6-b62e-bed8d86e3a54-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:31:14.111140 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.111110 5688 generic.go:356] "Generic (PLEG): container finished" podID="a178472e-7915-49a6-b62e-bed8d86e3a54" containerID="010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b" exitCode=0 Oct 31 19:31:14.111725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.111172 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" Oct 31 19:31:14.111725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.111222 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" event={"ID":"a178472e-7915-49a6-b62e-bed8d86e3a54","Type":"ContainerDied","Data":"010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b"} Oct 31 19:31:14.111725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.111312 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm" event={"ID":"a178472e-7915-49a6-b62e-bed8d86e3a54","Type":"ContainerDied","Data":"561dddd2a982cd32d53220f94ed557e98aa0816a0c3ab0c222ace4d5c288cc1a"} Oct 31 19:31:14.111725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.111347 5688 scope.go:122] "RemoveContainer" containerID="010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b" Oct 31 19:31:14.120460 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.120427 5688 scope.go:122] "RemoveContainer" containerID="010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b" Oct 31 19:31:14.120724 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:31:14.120705 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b\": container with ID starting with 010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b not found: ID does not exist" containerID="010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b" Oct 31 19:31:14.120786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.120733 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b"} err="failed to get container status \"010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b\": rpc error: code = NotFound desc = could not find container \"010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b\": container with ID starting with 010fb2eb2ff6fc4aac73531544fd2fe9500941c67030cca9b7ff4f9618c3264b not found: ID does not exist" Oct 31 19:31:14.274493 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.274465 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm"] Oct 31 19:31:14.324501 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:14.324471 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c97ce8-6644db6487-7zhlm"] Oct 31 19:31:15.566202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:15.566149 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a178472e-7915-49a6-b62e-bed8d86e3a54" path="/var/lib/kubelet/pods/a178472e-7915-49a6-b62e-bed8d86e3a54/volumes" Oct 31 19:31:20.193881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:20.193843 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:31:20.197928 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:20.197901 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:31:21.754996 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:21.754967 5688 scope.go:122] "RemoveContainer" containerID="0c66a7f361b65d62fc88b3597ef5aef2aab2cf8d99f008397551cc5db57c506a" Oct 31 19:31:24.903395 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:31:24.903361 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:31:54.904708 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:31:54.904674 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:31:57.080694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.080663 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f"] Oct 31 19:31:57.081113 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081101 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d8c6be1b-4020-4e57-97b7-95efa57d12fc" containerName="maas-api" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081114 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c6be1b-4020-4e57-97b7-95efa57d12fc" containerName="maas-api" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081125 5688 cpu_manager.go:414] "containerMap: removing container" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081131 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081140 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081144 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081150 5688 cpu_manager.go:414] "containerMap: removing container" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.081161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081154 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081165 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081169 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081174 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081179 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081204 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081209 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081217 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081222 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081227 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081232 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081237 5688 cpu_manager.go:414] "containerMap: removing container" podUID="a178472e-7915-49a6-b62e-bed8d86e3a54" containerName="payload-pre-processing" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081241 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a178472e-7915-49a6-b62e-bed8d86e3a54" containerName="payload-pre-processing" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081247 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081251 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081260 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081264 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081270 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081274 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081281 5688 cpu_manager.go:414] "containerMap: removing container" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081286 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081293 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081297 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081303 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081307 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081316 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081320 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081324 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081328 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081334 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081338 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081345 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081349 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081354 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081358 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081363 5688 cpu_manager.go:414] "containerMap: removing container" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081367 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081377 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081381 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081391 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.081434 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081395 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081465 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c6be1b-4020-4e57-97b7-95efa57d12fc" containerName="maas-api" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081472 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081478 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081483 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081489 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081493 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081499 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081504 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081509 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081514 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="a178472e-7915-49a6-b62e-bed8d86e3a54" containerName="payload-pre-processing" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081520 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081526 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081529 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081534 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081539 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081543 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081548 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2f39c2-09dc-4648-803d-9fd91d3b091d" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081553 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:31:57.082637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.081556 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e95b725-8577-4237-be04-fb88f12b264c" containerName="payload-processing" Oct 31 19:31:57.084677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.084661 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.143098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.143075 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f"] Oct 31 19:31:57.169965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.169935 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94kr\" (UniqueName: \"kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.170115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.169984 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.170219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.170164 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.186711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.186687 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:57.186833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.186775 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:31:57.270948 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.270921 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94kr\" (UniqueName: \"kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.271068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.270968 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.271068 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.271034 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.271816 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.271788 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.273481 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.273461 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.329255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.329228 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94kr\" (UniqueName: \"kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr\") pod \"maas-api-revoke-sub-a7976c60-ccp2f\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.397458 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.397399 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:31:57.767296 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:57.767272 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f"] Oct 31 19:31:57.769986 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:31:57.769943 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7406e4d_25f5_4733_a061_c24b4dab6491.slice/crio-3c8682f21a48d8a2d48b935048702915cc8360e8841a0f5aab7820c612d57567 WatchSource:0}: Error finding container 3c8682f21a48d8a2d48b935048702915cc8360e8841a0f5aab7820c612d57567: Status 404 returned error can't find the container with id 3c8682f21a48d8a2d48b935048702915cc8360e8841a0f5aab7820c612d57567 Oct 31 19:31:58.376494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:58.376402 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" event={"ID":"a7406e4d-25f5-4733-a061-c24b4dab6491","Type":"ContainerStarted","Data":"e9f565296dabf00ad2786a051848161cdbc12776b51d22ef63c1f8fd9d8d4a2a"} Oct 31 19:31:58.376494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:58.376475 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" event={"ID":"a7406e4d-25f5-4733-a061-c24b4dab6491","Type":"ContainerStarted","Data":"3c8682f21a48d8a2d48b935048702915cc8360e8841a0f5aab7820c612d57567"} Oct 31 19:31:58.540913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:58.540861 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" podStartSLOduration=1.540847986 podStartE2EDuration="1.540847986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:58.493573177 +0000 UTC m=+4239.095714321" watchObservedRunningTime="2026-10-31 19:31:58.540847986 +0000 UTC m=+4239.142989129" Oct 31 19:31:59.384155 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:59.384121 5688 generic.go:356] "Generic (PLEG): container finished" podID="a7406e4d-25f5-4733-a061-c24b4dab6491" containerID="e9f565296dabf00ad2786a051848161cdbc12776b51d22ef63c1f8fd9d8d4a2a" exitCode=0 Oct 31 19:31:59.384583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:31:59.384226 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" event={"ID":"a7406e4d-25f5-4733-a061-c24b4dab6491","Type":"ContainerDied","Data":"e9f565296dabf00ad2786a051848161cdbc12776b51d22ef63c1f8fd9d8d4a2a"} Oct 31 19:32:00.558658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.558632 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:32:00.702709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.702636 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca\") pod \"a7406e4d-25f5-4733-a061-c24b4dab6491\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " Oct 31 19:32:00.702709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.702675 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr\" (UniqueName: \"kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr\") pod \"a7406e4d-25f5-4733-a061-c24b4dab6491\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " Oct 31 19:32:00.702951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.702751 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca\") pod \"a7406e4d-25f5-4733-a061-c24b4dab6491\" (UID: \"a7406e4d-25f5-4733-a061-c24b4dab6491\") " Oct 31 19:32:00.703151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.703126 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca" pod "a7406e4d-25f5-4733-a061-c24b4dab6491" (UID: "a7406e4d-25f5-4733-a061-c24b4dab6491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:32:00.703451 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.703409 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:32:00.705098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.705058 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca" pod "a7406e4d-25f5-4733-a061-c24b4dab6491" (UID: "a7406e4d-25f5-4733-a061-c24b4dab6491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:32:00.705232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.705108 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr" pod "a7406e4d-25f5-4733-a061-c24b4dab6491" (UID: "a7406e4d-25f5-4733-a061-c24b4dab6491"). InnerVolumeSpecName "kube-api-access-m94kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:32:00.804872 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.804841 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7406e4d-25f5-4733-a061-c24b4dab6491-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:32:00.804872 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:00.804864 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94kr\" (UniqueName: \"kubernetes.io/projected/a7406e4d-25f5-4733-a061-c24b4dab6491-kube-api-access-m94kr\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:32:01.397658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:01.397628 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" Oct 31 19:32:01.397658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:01.397638 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f" event={"ID":"a7406e4d-25f5-4733-a061-c24b4dab6491","Type":"ContainerDied","Data":"3c8682f21a48d8a2d48b935048702915cc8360e8841a0f5aab7820c612d57567"} Oct 31 19:32:01.397885 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:32:01.397691 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8682f21a48d8a2d48b935048702915cc8360e8841a0f5aab7820c612d57567" Oct 31 19:32:24.904498 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:32:24.904461 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:32:54.904521 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:32:54.904483 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:33:06.864565 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.864526 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7"] Oct 31 19:33:06.865007 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.864997 5688 cpu_manager.go:414] "containerMap: removing container" podUID="a7406e4d-25f5-4733-a061-c24b4dab6491" containerName="revoke-keys" Oct 31 19:33:06.865062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865008 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7406e4d-25f5-4733-a061-c24b4dab6491" containerName="revoke-keys" Oct 31 19:33:06.865062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865032 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:33:06.865062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865037 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:33:06.865062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865056 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:33:06.865062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865061 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865151 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7406e4d-25f5-4733-a061-c24b4dab6491" containerName="revoke-keys" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865159 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="87498025-815e-4cec-b6db-a79423447931" containerName="cleanup" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865168 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f134f9d-1862-48db-95d4-843d1a967dd9" containerName="istio-proxy" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865175 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c7051a-1c9b-457b-9bde-b83e3878f2c2" containerName="cleanup" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865198 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b51bfe4-eaad-401d-924e-636f65470576" containerName="cleanup" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865203 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59eba41-82be-4721-8082-212b6df76bed" containerName="cleanup" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865209 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0c8e9d-ef25-4ecc-b8d0-0b2b12fc6de1" containerName="cleanup" Oct 31 19:33:06.865265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.865215 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0980939e-1112-4300-91d8-a4e9437df583" containerName="cleanup" Oct 31 19:33:06.868461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.868443 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:06.925535 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.925497 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7"] Oct 31 19:33:06.929212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.929153 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:06.929357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.929263 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:06.929357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.929310 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlb5\" (UniqueName: \"kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:06.972066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.972037 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:33:06.972278 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:06.972090 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:33:07.030648 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.030614 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.030820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.030659 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.030820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.030692 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlb5\" (UniqueName: \"kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.031398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.031374 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.033157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.033125 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.088302 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.088270 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlb5\" (UniqueName: \"kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5\") pod \"maas-api-revoke-sub-762a48bd-2gkv7\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.181932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.181852 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:07.555821 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.555796 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7"] Oct 31 19:33:07.558385 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:33:07.558357 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2614e9aa_3704_4cd5_aac8_7b5f58071651.slice/crio-a258d8f2d498ebbd26912b08f5ea289356c08608e007bf496ae13cbf88a711c8 WatchSource:0}: Error finding container a258d8f2d498ebbd26912b08f5ea289356c08608e007bf496ae13cbf88a711c8: Status 404 returned error can't find the container with id a258d8f2d498ebbd26912b08f5ea289356c08608e007bf496ae13cbf88a711c8 Oct 31 19:33:07.766749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:07.766714 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" event={"ID":"2614e9aa-3704-4cd5-aac8-7b5f58071651","Type":"ContainerStarted","Data":"a258d8f2d498ebbd26912b08f5ea289356c08608e007bf496ae13cbf88a711c8"} Oct 31 19:33:08.773839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:08.773804 5688 generic.go:356] "Generic (PLEG): container finished" podID="2614e9aa-3704-4cd5-aac8-7b5f58071651" containerID="d248e6106601c4b25b4f4fc404995af598506881585bf250968afadc523f9bec" exitCode=0 Oct 31 19:33:08.774255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:08.773867 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" event={"ID":"2614e9aa-3704-4cd5-aac8-7b5f58071651","Type":"ContainerDied","Data":"d248e6106601c4b25b4f4fc404995af598506881585bf250968afadc523f9bec"} Oct 31 19:33:09.919693 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.919669 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:09.961524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.961500 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5\" (UniqueName: \"kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5\") pod \"2614e9aa-3704-4cd5-aac8-7b5f58071651\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " Oct 31 19:33:09.961669 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.961556 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca\") pod \"2614e9aa-3704-4cd5-aac8-7b5f58071651\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " Oct 31 19:33:09.961669 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.961598 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca\") pod \"2614e9aa-3704-4cd5-aac8-7b5f58071651\" (UID: \"2614e9aa-3704-4cd5-aac8-7b5f58071651\") " Oct 31 19:33:09.961918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.961892 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca" pod "2614e9aa-3704-4cd5-aac8-7b5f58071651" (UID: "2614e9aa-3704-4cd5-aac8-7b5f58071651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:09.963656 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.963631 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca" pod "2614e9aa-3704-4cd5-aac8-7b5f58071651" (UID: "2614e9aa-3704-4cd5-aac8-7b5f58071651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:09.963803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:09.963786 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5" pod "2614e9aa-3704-4cd5-aac8-7b5f58071651" (UID: "2614e9aa-3704-4cd5-aac8-7b5f58071651"). InnerVolumeSpecName "kube-api-access-pwlb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:10.063396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:10.063312 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwlb5\" (UniqueName: \"kubernetes.io/projected/2614e9aa-3704-4cd5-aac8-7b5f58071651-kube-api-access-pwlb5\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:10.063396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:10.063353 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:10.063396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:10.063370 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2614e9aa-3704-4cd5-aac8-7b5f58071651-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:10.786918 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:10.786884 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" event={"ID":"2614e9aa-3704-4cd5-aac8-7b5f58071651","Type":"ContainerDied","Data":"a258d8f2d498ebbd26912b08f5ea289356c08608e007bf496ae13cbf88a711c8"} Oct 31 19:33:10.787128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:10.786923 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7" Oct 31 19:33:10.787128 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:10.786935 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a258d8f2d498ebbd26912b08f5ea289356c08608e007bf496ae13cbf88a711c8" Oct 31 19:33:20.243961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.243924 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth"] Oct 31 19:33:20.244590 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.244121 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" podUID="15db64a1-85fd-48ee-a80c-391fd786d58d" containerName="maas-api" containerID="cri-o://b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb" gracePeriod=30 Oct 31 19:33:20.579400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.579370 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:33:20.701112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.700482 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6\" (UniqueName: \"kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6\") pod \"15db64a1-85fd-48ee-a80c-391fd786d58d\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " Oct 31 19:33:20.701112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.700628 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\") pod \"15db64a1-85fd-48ee-a80c-391fd786d58d\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " Oct 31 19:33:20.701112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.700678 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca\") pod \"15db64a1-85fd-48ee-a80c-391fd786d58d\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " Oct 31 19:33:20.701112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.700830 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs\") pod \"15db64a1-85fd-48ee-a80c-391fd786d58d\" (UID: \"15db64a1-85fd-48ee-a80c-391fd786d58d\") " Oct 31 19:33:20.702929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.702884 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca" pod "15db64a1-85fd-48ee-a80c-391fd786d58d" (UID: "15db64a1-85fd-48ee-a80c-391fd786d58d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:20.706373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.706349 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6" pod "15db64a1-85fd-48ee-a80c-391fd786d58d" (UID: "15db64a1-85fd-48ee-a80c-391fd786d58d"). InnerVolumeSpecName "kube-api-access-q6gs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:20.708796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.708768 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs" pod "15db64a1-85fd-48ee-a80c-391fd786d58d" (UID: "15db64a1-85fd-48ee-a80c-391fd786d58d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:20.709333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.709302 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls" pod "15db64a1-85fd-48ee-a80c-391fd786d58d" (UID: "15db64a1-85fd-48ee-a80c-391fd786d58d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:20.802605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.802521 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:20.802605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.802550 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6gs6\" (UniqueName: \"kubernetes.io/projected/15db64a1-85fd-48ee-a80c-391fd786d58d-kube-api-access-q6gs6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:20.802605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.802562 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15db64a1-85fd-48ee-a80c-391fd786d58d-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:20.802605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.802574 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15db64a1-85fd-48ee-a80c-391fd786d58d-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:33:20.847694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.847653 5688 generic.go:356] "Generic (PLEG): container finished" podID="15db64a1-85fd-48ee-a80c-391fd786d58d" containerID="b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb" exitCode=0 Oct 31 19:33:20.847906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.847735 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" event={"ID":"15db64a1-85fd-48ee-a80c-391fd786d58d","Type":"ContainerDied","Data":"b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb"} Oct 31 19:33:20.847906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.847753 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" Oct 31 19:33:20.847906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.847785 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth" event={"ID":"15db64a1-85fd-48ee-a80c-391fd786d58d","Type":"ContainerDied","Data":"c1cc046db4c8cfea8a4bd315c3c9ef7b03255ece9c147899a1376a6eb4feab05"} Oct 31 19:33:20.847906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.847812 5688 scope.go:122] "RemoveContainer" containerID="b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb" Oct 31 19:33:20.858969 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.858947 5688 scope.go:122] "RemoveContainer" containerID="b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb" Oct 31 19:33:20.859298 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:33:20.859273 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb\": container with ID starting with b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb not found: ID does not exist" containerID="b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb" Oct 31 19:33:20.859345 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:20.859306 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb"} err="failed to get container status \"b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb\": rpc error: code = NotFound desc = could not find container \"b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb\": container with ID starting with b45575ff9767e3a213d71d06904c38baa7a14404d0da8698c54799274c133fcb not found: ID does not exist" Oct 31 19:33:21.011003 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:21.010967 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth"] Oct 31 19:33:21.014559 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:21.014532 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fdd317-6c9d6dfbcc-bwxth"] Oct 31 19:33:21.566075 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:21.566040 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15db64a1-85fd-48ee-a80c-391fd786d58d" path="/var/lib/kubelet/pods/15db64a1-85fd-48ee-a80c-391fd786d58d/volumes" Oct 31 19:33:24.904799 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:33:24.904763 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:33:46.701571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.701539 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn"] Oct 31 19:33:46.702014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.701990 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2614e9aa-3704-4cd5-aac8-7b5f58071651" containerName="revoke-keys" Oct 31 19:33:46.702014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.702001 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2614e9aa-3704-4cd5-aac8-7b5f58071651" containerName="revoke-keys" Oct 31 19:33:46.702090 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.702022 5688 cpu_manager.go:414] "containerMap: removing container" podUID="15db64a1-85fd-48ee-a80c-391fd786d58d" containerName="maas-api" Oct 31 19:33:46.702090 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.702027 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15db64a1-85fd-48ee-a80c-391fd786d58d" containerName="maas-api" Oct 31 19:33:46.702158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.702131 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="15db64a1-85fd-48ee-a80c-391fd786d58d" containerName="maas-api" Oct 31 19:33:46.702158 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.702146 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2614e9aa-3704-4cd5-aac8-7b5f58071651" containerName="revoke-keys" Oct 31 19:33:46.705658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.705633 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:46.829234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.823461 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-049c8941\"" Oct 31 19:33:46.829234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.823556 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-049c8941-dockercfg-mc7rl\"" Oct 31 19:33:46.831944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.829834 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn"] Oct 31 19:33:46.842801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.839002 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdl9n\" (UniqueName: \"kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n\") pod \"payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:46.842801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.839137 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume\") pod \"payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:46.877062 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.877024 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm"] Oct 31 19:33:46.881232 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.881206 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:46.918794 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.918767 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm"] Oct 31 19:33:46.940773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.940732 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:46.940961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.940815 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdl9n\" (UniqueName: \"kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n\") pod \"payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:46.940961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.940912 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlgk\" (UniqueName: \"kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk\") pod \"payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:46.941055 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.941036 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume\") pod \"payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:46.941769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.941751 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume\") pod \"payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:46.998856 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:46.998827 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdl9n\" (UniqueName: \"kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n\") pod \"payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:47.019594 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.019569 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:47.041968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.041938 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlgk\" (UniqueName: \"kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk\") pod \"payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:47.042121 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.042056 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:47.042938 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.042913 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:47.100123 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.100091 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlgk\" (UniqueName: \"kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk\") pod \"payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:47.195381 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.195342 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:47.410555 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.410530 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn"] Oct 31 19:33:47.413064 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:33:47.413028 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982d98ea_012d_4340_8e07_239f67af16eb.slice/crio-391459d821f19297b4e68fc8cfbd4dbad361241231be619605304af8ff07fafd WatchSource:0}: Error finding container 391459d821f19297b4e68fc8cfbd4dbad361241231be619605304af8ff07fafd: Status 404 returned error can't find the container with id 391459d821f19297b4e68fc8cfbd4dbad361241231be619605304af8ff07fafd Oct 31 19:33:47.576543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:47.576406 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm"] Oct 31 19:33:47.578847 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:33:47.578822 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0c77dc_db9c_4aab_a07b_e0f4c6bdd87c.slice/crio-fa1f7801c5dee1e8322d289b071e0ba5303f05d23a99d2795bf3bbaea619be9e WatchSource:0}: Error finding container fa1f7801c5dee1e8322d289b071e0ba5303f05d23a99d2795bf3bbaea619be9e: Status 404 returned error can't find the container with id fa1f7801c5dee1e8322d289b071e0ba5303f05d23a99d2795bf3bbaea619be9e Oct 31 19:33:48.011473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.011421 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" event={"ID":"982d98ea-012d-4340-8e07-239f67af16eb","Type":"ContainerStarted","Data":"fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38"} Oct 31 19:33:48.011933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.011597 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:48.011933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.011645 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" event={"ID":"982d98ea-012d-4340-8e07-239f67af16eb","Type":"ContainerStarted","Data":"391459d821f19297b4e68fc8cfbd4dbad361241231be619605304af8ff07fafd"} Oct 31 19:33:48.013066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.013043 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" event={"ID":"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c","Type":"ContainerStarted","Data":"f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165"} Oct 31 19:33:48.013205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.013079 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" event={"ID":"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c","Type":"ContainerStarted","Data":"fa1f7801c5dee1e8322d289b071e0ba5303f05d23a99d2795bf3bbaea619be9e"} Oct 31 19:33:48.013205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.013097 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:33:48.125971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.125903 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" podStartSLOduration=2.125887206 podStartE2EDuration="2.125887206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:48.125742041 +0000 UTC m=+4348.727883189" watchObservedRunningTime="2026-10-31 19:33:48.125887206 +0000 UTC m=+4348.728028349" Oct 31 19:33:48.255266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:48.255144 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" podStartSLOduration=2.255122283 podStartE2EDuration="2.255122283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:48.253795037 +0000 UTC m=+4348.855936206" watchObservedRunningTime="2026-10-31 19:33:48.255122283 +0000 UTC m=+4348.857263429" Oct 31 19:33:54.937478 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:33:54.937440 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:33:59.023120 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:59.023077 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:33:59.023626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:33:59.023156 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:34:24.905982 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:34:24.905943 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:34:54.905317 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:34:54.905282 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:35:24.904617 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:35:24.904582 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:35:54.904314 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:35:54.904275 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:36:20.231023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:20.230981 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:36:20.235375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:20.235356 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:36:24.904489 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:36:24.904454 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:36:32.113470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.113381 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g"] Oct 31 19:36:32.117336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.117316 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.169680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.169642 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.169680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.169691 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.169944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.169876 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.169944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.169930 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.170069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.170012 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.170069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.170054 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggb7\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.170219 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.170097 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.170348 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.170329 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.170443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.170344 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0f794f-openshift-default-dockercfg-pgj48\"" Oct 31 19:36:32.170443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.170411 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.223333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.223300 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g"] Oct 31 19:36:32.291210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.281895 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.281963 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.282044 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.282134 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.291304 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.291366 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.291424 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.291469 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.291537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.291496 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggb7\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.298703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.298665 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.299056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.299028 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.299499 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.299473 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.299732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.299712 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.299837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.299814 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.300909 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.300851 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.304511 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.302405 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.349756 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.349726 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggb7\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.377956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.377708 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token\") pod \"e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.434036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.434000 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:32.871573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.871544 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g"] Oct 31 19:36:32.872663 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:36:32.872637 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087b9573_9e58_42ae_a315_80911d84e069.slice/crio-40b00adbe9ffdf6523c345dc5b21c26f0a863d7166dacb628a260ef7b16a831f WatchSource:0}: Error finding container 40b00adbe9ffdf6523c345dc5b21c26f0a863d7166dacb628a260ef7b16a831f: Status 404 returned error can't find the container with id 40b00adbe9ffdf6523c345dc5b21c26f0a863d7166dacb628a260ef7b16a831f Oct 31 19:36:32.875034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.874999 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:36:32.875108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.875069 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:36:32.875108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.875098 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:36:32.933759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:32.933731 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" event={"ID":"087b9573-9e58-42ae-a315-80911d84e069","Type":"ContainerStarted","Data":"40b00adbe9ffdf6523c345dc5b21c26f0a863d7166dacb628a260ef7b16a831f"} Oct 31 19:36:33.940786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:33.940751 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" event={"ID":"087b9573-9e58-42ae-a315-80911d84e069","Type":"ContainerStarted","Data":"978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62"} Oct 31 19:36:34.167443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:34.167385 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podStartSLOduration=2.16736809 podStartE2EDuration="2.16736809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:34.11522386 +0000 UTC m=+4514.717365003" watchObservedRunningTime="2026-10-31 19:36:34.16736809 +0000 UTC m=+4514.769509237" Oct 31 19:36:34.434417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:34.434387 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:34.441111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:34.441086 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:34.946083 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:34.946054 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:34.947065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:34.947046 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:36:40.809175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.809146 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g"] Oct 31 19:36:40.813868 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.813838 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:40.869937 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.869908 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g"] Oct 31 19:36:40.917204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.917156 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0f794f\"" Oct 31 19:36:40.917338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.917230 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0f794f-dockercfg-ng4dt\"" Oct 31 19:36:40.983461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.983430 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwqf\" (UniqueName: \"kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf\") pod \"payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:40.983603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:40.983479 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:41.084443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.084369 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwqf\" (UniqueName: \"kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf\") pod \"payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:41.084443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.084414 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:41.085100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.085076 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:41.145052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.144359 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwqf\" (UniqueName: \"kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf\") pod \"payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:41.429680 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.429606 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:41.804563 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.804533 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g"] Oct 31 19:36:41.806446 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:36:41.806415 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5273ca_0444_433d_9814_0af5f263fe67.slice/crio-6bdf0c4fecdb370bde1732e986237289c48a8c0eaa73d10c66b183d23b213dfa WatchSource:0}: Error finding container 6bdf0c4fecdb370bde1732e986237289c48a8c0eaa73d10c66b183d23b213dfa: Status 404 returned error can't find the container with id 6bdf0c4fecdb370bde1732e986237289c48a8c0eaa73d10c66b183d23b213dfa Oct 31 19:36:41.985902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.985875 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" event={"ID":"0a5273ca-0444-433d-9814-0af5f263fe67","Type":"ContainerStarted","Data":"21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046"} Oct 31 19:36:41.986271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.985921 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" event={"ID":"0a5273ca-0444-433d-9814-0af5f263fe67","Type":"ContainerStarted","Data":"6bdf0c4fecdb370bde1732e986237289c48a8c0eaa73d10c66b183d23b213dfa"} Oct 31 19:36:41.986271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:41.985964 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:42.103430 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:42.103344 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" podStartSLOduration=2.103327061 podStartE2EDuration="2.103327061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:42.101521506 +0000 UTC m=+4522.703662649" watchObservedRunningTime="2026-10-31 19:36:42.103327061 +0000 UTC m=+4522.705468205" Oct 31 19:36:52.994835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:36:52.994809 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:36:54.904923 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:36:54.904888 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:37:01.046133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:01.046101 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f"] Oct 31 19:37:01.096138 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:01.096105 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7976c60-ccp2f"] Oct 31 19:37:01.565525 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:01.565487 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7406e4d-25f5-4733-a061-c24b4dab6491" path="/var/lib/kubelet/pods/a7406e4d-25f5-4733-a061-c24b4dab6491/volumes" Oct 31 19:37:24.904053 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:37:24.904015 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:37:45.476117 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:45.476080 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn"] Oct 31 19:37:45.580660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:45.580627 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm"] Oct 31 19:37:50.498450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.498355 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" podUID="982d98ea-012d-4340-8e07-239f67af16eb" containerName="payload-processing" containerID="cri-o://fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38" gracePeriod=25 Oct 31 19:37:50.581296 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.581227 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" podUID="5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" containerName="payload-pre-processing" containerID="cri-o://f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165" gracePeriod=25 Oct 31 19:37:50.844472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.844444 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:37:50.868280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.868260 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:37:50.944745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.944711 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk\" (UniqueName: \"kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk\") pod \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " Oct 31 19:37:50.944951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.944830 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n\" (UniqueName: \"kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n\") pod \"982d98ea-012d-4340-8e07-239f67af16eb\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " Oct 31 19:37:50.945024 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.944951 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume\") pod \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\" (UID: \"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c\") " Oct 31 19:37:50.945073 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.945034 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume\") pod \"982d98ea-012d-4340-8e07-239f67af16eb\" (UID: \"982d98ea-012d-4340-8e07-239f67af16eb\") " Oct 31 19:37:50.945478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.945362 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume" pod "5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" (UID: "5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:37:50.945478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.945415 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume" pod "982d98ea-012d-4340-8e07-239f67af16eb" (UID: "982d98ea-012d-4340-8e07-239f67af16eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:37:50.945478 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.945451 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:37:50.947643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.947624 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n" pod "982d98ea-012d-4340-8e07-239f67af16eb" (UID: "982d98ea-012d-4340-8e07-239f67af16eb"). InnerVolumeSpecName "kube-api-access-kdl9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:37:50.947721 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:50.947651 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk" pod "5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" (UID: "5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c"). InnerVolumeSpecName "kube-api-access-vnlgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:37:51.046602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.046532 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/982d98ea-012d-4340-8e07-239f67af16eb-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:37:51.046602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.046563 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnlgk\" (UniqueName: \"kubernetes.io/projected/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c-kube-api-access-vnlgk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:37:51.046602 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.046572 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdl9n\" (UniqueName: \"kubernetes.io/projected/982d98ea-012d-4340-8e07-239f67af16eb-kube-api-access-kdl9n\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:37:51.383568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.383483 5688 generic.go:356] "Generic (PLEG): container finished" podID="5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" containerID="f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165" exitCode=0 Oct 31 19:37:51.383568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.383529 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" Oct 31 19:37:51.383790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.383555 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" event={"ID":"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c","Type":"ContainerDied","Data":"f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165"} Oct 31 19:37:51.383790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.383649 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm" event={"ID":"5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c","Type":"ContainerDied","Data":"fa1f7801c5dee1e8322d289b071e0ba5303f05d23a99d2795bf3bbaea619be9e"} Oct 31 19:37:51.383790 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.383683 5688 scope.go:122] "RemoveContainer" containerID="f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165" Oct 31 19:37:51.385260 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.385236 5688 generic.go:356] "Generic (PLEG): container finished" podID="982d98ea-012d-4340-8e07-239f67af16eb" containerID="fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38" exitCode=0 Oct 31 19:37:51.385385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.385290 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" Oct 31 19:37:51.385385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.385296 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" event={"ID":"982d98ea-012d-4340-8e07-239f67af16eb","Type":"ContainerDied","Data":"fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38"} Oct 31 19:37:51.385385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.385334 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn" event={"ID":"982d98ea-012d-4340-8e07-239f67af16eb","Type":"ContainerDied","Data":"391459d821f19297b4e68fc8cfbd4dbad361241231be619605304af8ff07fafd"} Oct 31 19:37:51.395294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.395266 5688 scope.go:122] "RemoveContainer" containerID="f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165" Oct 31 19:37:51.395657 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:37:51.395636 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165\": container with ID starting with f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165 not found: ID does not exist" containerID="f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165" Oct 31 19:37:51.395744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.395664 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165"} err="failed to get container status \"f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165\": rpc error: code = NotFound desc = could not find container \"f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165\": container with ID starting with f556b8d65f7662f07858df3281462ac1515935c7a98ab42d890c86ca5dc86165 not found: ID does not exist" Oct 31 19:37:51.395744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.395684 5688 scope.go:122] "RemoveContainer" containerID="fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38" Oct 31 19:37:51.405093 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.405075 5688 scope.go:122] "RemoveContainer" containerID="fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38" Oct 31 19:37:51.405355 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:37:51.405337 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38\": container with ID starting with fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38 not found: ID does not exist" containerID="fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38" Oct 31 19:37:51.405417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.405363 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38"} err="failed to get container status \"fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38\": rpc error: code = NotFound desc = could not find container \"fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38\": container with ID starting with fca6985584a2cd360b1421611c06bf118ca6b1200ba16b86b1b61a6139c68e38 not found: ID does not exist" Oct 31 19:37:51.546727 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.546699 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm"] Oct 31 19:37:51.550483 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.550458 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049c8941-76f469d6c5-8skdm"] Oct 31 19:37:51.565785 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.565762 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" path="/var/lib/kubelet/pods/5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c/volumes" Oct 31 19:37:51.658526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.658471 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn"] Oct 31 19:37:51.662567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:51.662527 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049c8941-dbcb64bb7-gnsvn"] Oct 31 19:37:53.566036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:53.566004 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982d98ea-012d-4340-8e07-239f67af16eb" path="/var/lib/kubelet/pods/982d98ea-012d-4340-8e07-239f67af16eb/volumes" Oct 31 19:37:54.902817 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:37:54.902786 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:37:56.606126 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:56.606096 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm"] Oct 31 19:37:56.707102 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:37:56.707067 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5"] Oct 31 19:38:01.606880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:01.606785 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerName="payload-processing" containerID="cri-o://7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30" gracePeriod=25 Oct 31 19:38:01.707586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:01.707548 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" podUID="03045557-3751-45d5-a215-3563cd36b8a2" containerName="payload-pre-processing" containerID="cri-o://00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878" gracePeriod=25 Oct 31 19:38:03.876906 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:03.876871 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" podUID="03045557-3751-45d5-a215-3563cd36b8a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:38:03.877329 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:03.876871 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:38:11.057595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:11.057563 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7"] Oct 31 19:38:11.106871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:11.106842 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a48bd-2gkv7"] Oct 31 19:38:11.566403 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:11.566374 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2614e9aa-3704-4cd5-aac8-7b5f58071651" path="/var/lib/kubelet/pods/2614e9aa-3704-4cd5-aac8-7b5f58071651/volumes" Oct 31 19:38:13.877561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:13.877519 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:38:13.877561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:13.877545 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" podUID="03045557-3751-45d5-a215-3563cd36b8a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:38:21.874131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:21.873905 5688 scope.go:122] "RemoveContainer" containerID="e9f565296dabf00ad2786a051848161cdbc12776b51d22ef63c1f8fd9d8d4a2a" Oct 31 19:38:21.962950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:21.962910 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:38:22.018083 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.018057 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:38:22.098722 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.098694 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f\" (UniqueName: \"kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f\") pod \"d0a88d47-278b-4f21-9831-fb4bd1359194\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " Oct 31 19:38:22.098894 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.098863 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume\") pod \"d0a88d47-278b-4f21-9831-fb4bd1359194\" (UID: \"d0a88d47-278b-4f21-9831-fb4bd1359194\") " Oct 31 19:38:22.099301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.099276 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume" pod "d0a88d47-278b-4f21-9831-fb4bd1359194" (UID: "d0a88d47-278b-4f21-9831-fb4bd1359194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:22.101289 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.101260 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f" pod "d0a88d47-278b-4f21-9831-fb4bd1359194" (UID: "d0a88d47-278b-4f21-9831-fb4bd1359194"). InnerVolumeSpecName "kube-api-access-t989f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:22.200250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.199928 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume\") pod \"03045557-3751-45d5-a215-3563cd36b8a2\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " Oct 31 19:38:22.200250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.200027 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd\" (UniqueName: \"kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd\") pod \"03045557-3751-45d5-a215-3563cd36b8a2\" (UID: \"03045557-3751-45d5-a215-3563cd36b8a2\") " Oct 31 19:38:22.200477 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.200341 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume" pod "03045557-3751-45d5-a215-3563cd36b8a2" (UID: "03045557-3751-45d5-a215-3563cd36b8a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:22.200477 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.200473 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a88d47-278b-4f21-9831-fb4bd1359194-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:22.200584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.200487 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03045557-3751-45d5-a215-3563cd36b8a2-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:22.200584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.200497 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t989f\" (UniqueName: \"kubernetes.io/projected/d0a88d47-278b-4f21-9831-fb4bd1359194-kube-api-access-t989f\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:22.202754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.202726 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd" pod "03045557-3751-45d5-a215-3563cd36b8a2" (UID: "03045557-3751-45d5-a215-3563cd36b8a2"). InnerVolumeSpecName "kube-api-access-j9zkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:22.301829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.301798 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9zkd\" (UniqueName: \"kubernetes.io/projected/03045557-3751-45d5-a215-3563cd36b8a2-kube-api-access-j9zkd\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:22.599459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.599421 5688 generic.go:356] "Generic (PLEG): container finished" podID="03045557-3751-45d5-a215-3563cd36b8a2" containerID="00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878" exitCode=0 Oct 31 19:38:22.599652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.599463 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" Oct 31 19:38:22.599652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.599503 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" event={"ID":"03045557-3751-45d5-a215-3563cd36b8a2","Type":"ContainerDied","Data":"00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878"} Oct 31 19:38:22.599652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.599552 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5" event={"ID":"03045557-3751-45d5-a215-3563cd36b8a2","Type":"ContainerDied","Data":"e1cb71410e925f354bad09f980610e732a5ac099c7ba944ec808a1b98d65dafc"} Oct 31 19:38:22.599652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.599574 5688 scope.go:122] "RemoveContainer" containerID="00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878" Oct 31 19:38:22.600936 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.600914 5688 generic.go:356] "Generic (PLEG): container finished" podID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerID="7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30" exitCode=0 Oct 31 19:38:22.601009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.600955 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" Oct 31 19:38:22.601009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.600977 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" event={"ID":"d0a88d47-278b-4f21-9831-fb4bd1359194","Type":"ContainerDied","Data":"7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30"} Oct 31 19:38:22.601094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.601016 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm" event={"ID":"d0a88d47-278b-4f21-9831-fb4bd1359194","Type":"ContainerDied","Data":"6a35f2142293d82ff641baa33ddc30b9d71d8b9fef2cafc2f7b1f63fbe5b5e48"} Oct 31 19:38:22.610800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.610786 5688 scope.go:122] "RemoveContainer" containerID="00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878" Oct 31 19:38:22.611243 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:38:22.611096 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878\": container with ID starting with 00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878 not found: ID does not exist" containerID="00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878" Oct 31 19:38:22.611243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.611123 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878"} err="failed to get container status \"00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878\": rpc error: code = NotFound desc = could not find container \"00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878\": container with ID starting with 00c093f0b0abbe373695ed2e15ad07700874d92e9bab119d7a731ae18e800878 not found: ID does not exist" Oct 31 19:38:22.611243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.611139 5688 scope.go:122] "RemoveContainer" containerID="7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30" Oct 31 19:38:22.631616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.631596 5688 scope.go:122] "RemoveContainer" containerID="7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30" Oct 31 19:38:22.631875 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:38:22.631856 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30\": container with ID starting with 7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30 not found: ID does not exist" containerID="7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30" Oct 31 19:38:22.631941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.631887 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30"} err="failed to get container status \"7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30\": rpc error: code = NotFound desc = could not find container \"7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30\": container with ID starting with 7751342914c5046bc94a5dadbeb19de1644b721a0fda1604f35c91ae8118cf30 not found: ID does not exist" Oct 31 19:38:22.763916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.763890 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5"] Oct 31 19:38:22.813854 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.813828 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4731a-cb4f97875-ndgz5"] Oct 31 19:38:22.923107 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.923030 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm"] Oct 31 19:38:22.929521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.929493 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4731a-95dbb8b77-7j5gm"] Oct 31 19:38:22.993077 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993047 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5"] Oct 31 19:38:22.993575 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993561 5688 cpu_manager.go:414] "containerMap: removing container" podUID="03045557-3751-45d5-a215-3563cd36b8a2" containerName="payload-pre-processing" Oct 31 19:38:22.993637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993578 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03045557-3751-45d5-a215-3563cd36b8a2" containerName="payload-pre-processing" Oct 31 19:38:22.993637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993599 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerName="payload-processing" Oct 31 19:38:22.993637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993606 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerName="payload-processing" Oct 31 19:38:22.993637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993621 5688 cpu_manager.go:414] "containerMap: removing container" podUID="982d98ea-012d-4340-8e07-239f67af16eb" containerName="payload-processing" Oct 31 19:38:22.993637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993626 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982d98ea-012d-4340-8e07-239f67af16eb" containerName="payload-processing" Oct 31 19:38:22.993825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993643 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" containerName="payload-pre-processing" Oct 31 19:38:22.993825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993648 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" containerName="payload-pre-processing" Oct 31 19:38:22.993825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993743 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0c77dc-db9c-4aab-a07b-e0f4c6bdd87c" containerName="payload-pre-processing" Oct 31 19:38:22.993825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993762 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="03045557-3751-45d5-a215-3563cd36b8a2" containerName="payload-pre-processing" Oct 31 19:38:22.993825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993771 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" containerName="payload-processing" Oct 31 19:38:22.993825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.993781 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="982d98ea-012d-4340-8e07-239f67af16eb" containerName="payload-processing" Oct 31 19:38:22.998522 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:22.998497 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.051272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.051235 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:38:23.051272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.051268 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:23.057007 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.053927 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5"] Oct 31 19:38:23.108707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.108675 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.108880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.108825 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6ps\" (UniqueName: \"kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.108977 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.108962 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.210454 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.210369 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.210454 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.210424 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.210691 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.210526 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6ps\" (UniqueName: \"kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.211121 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.211101 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.212910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.212888 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.268464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.268429 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6ps\" (UniqueName: \"kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps\") pod \"maas-api-revoke-sub-cd966ca1-8wgx5\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.312169 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.312122 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:23.567192 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.567151 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03045557-3751-45d5-a215-3563cd36b8a2" path="/var/lib/kubelet/pods/03045557-3751-45d5-a215-3563cd36b8a2/volumes" Oct 31 19:38:23.567578 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.567565 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a88d47-278b-4f21-9831-fb4bd1359194" path="/var/lib/kubelet/pods/d0a88d47-278b-4f21-9831-fb4bd1359194/volumes" Oct 31 19:38:23.693671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:23.693626 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5"] Oct 31 19:38:23.697222 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:38:23.697175 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c09fd91_63a7_4699_a611_61e6e7038a5e.slice/crio-7e142f98e59be4db5e5bf4f217bd9b77ca05c57ab8d6c61bb2abeb4fd18c160d WatchSource:0}: Error finding container 7e142f98e59be4db5e5bf4f217bd9b77ca05c57ab8d6c61bb2abeb4fd18c160d: Status 404 returned error can't find the container with id 7e142f98e59be4db5e5bf4f217bd9b77ca05c57ab8d6c61bb2abeb4fd18c160d Oct 31 19:38:24.617196 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:24.617161 5688 generic.go:356] "Generic (PLEG): container finished" podID="6c09fd91-63a7-4699-a611-61e6e7038a5e" containerID="4a8155457dc8af875d25c48df5071bf1eee3925ec0a178e92ccf1b47a3d85fa5" exitCode=0 Oct 31 19:38:24.617516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:24.617216 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" event={"ID":"6c09fd91-63a7-4699-a611-61e6e7038a5e","Type":"ContainerDied","Data":"4a8155457dc8af875d25c48df5071bf1eee3925ec0a178e92ccf1b47a3d85fa5"} Oct 31 19:38:24.617516 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:24.617271 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" event={"ID":"6c09fd91-63a7-4699-a611-61e6e7038a5e","Type":"ContainerStarted","Data":"7e142f98e59be4db5e5bf4f217bd9b77ca05c57ab8d6c61bb2abeb4fd18c160d"} Oct 31 19:38:24.904779 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:38:24.904692 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:38:25.623933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.623907 5688 generic.go:356] "Generic (PLEG): container finished" podID="087b9573-9e58-42ae-a315-80911d84e069" containerID="978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62" exitCode=0 Oct 31 19:38:25.624347 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.623987 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" event={"ID":"087b9573-9e58-42ae-a315-80911d84e069","Type":"ContainerDied","Data":"978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62"} Oct 31 19:38:25.624734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.624577 5688 scope.go:122] "RemoveContainer" containerID="978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62" Oct 31 19:38:25.626056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.626026 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:38:25.626151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.626109 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:38:25.626151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.626144 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:38:25.866249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.865641 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:25.942216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.939817 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca\") pod \"6c09fd91-63a7-4699-a611-61e6e7038a5e\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " Oct 31 19:38:25.942216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.939875 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca\") pod \"6c09fd91-63a7-4699-a611-61e6e7038a5e\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " Oct 31 19:38:25.960107 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.955114 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps\" (UniqueName: \"kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps\") pod \"6c09fd91-63a7-4699-a611-61e6e7038a5e\" (UID: \"6c09fd91-63a7-4699-a611-61e6e7038a5e\") " Oct 31 19:38:25.960107 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.959013 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca" pod "6c09fd91-63a7-4699-a611-61e6e7038a5e" (UID: "6c09fd91-63a7-4699-a611-61e6e7038a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:25.964354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.964320 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca" pod "6c09fd91-63a7-4699-a611-61e6e7038a5e" (UID: "6c09fd91-63a7-4699-a611-61e6e7038a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:38:25.968054 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:25.968008 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps" pod "6c09fd91-63a7-4699-a611-61e6e7038a5e" (UID: "6c09fd91-63a7-4699-a611-61e6e7038a5e"). InnerVolumeSpecName "kube-api-access-pz6ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:26.069212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.063923 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz6ps\" (UniqueName: \"kubernetes.io/projected/6c09fd91-63a7-4699-a611-61e6e7038a5e-kube-api-access-pz6ps\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:26.069212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.063965 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:26.069212 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.063980 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c09fd91-63a7-4699-a611-61e6e7038a5e-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:26.658643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.658598 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" event={"ID":"087b9573-9e58-42ae-a315-80911d84e069","Type":"ContainerStarted","Data":"74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882"} Oct 31 19:38:26.660459 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.660428 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" Oct 31 19:38:26.660684 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.660427 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5" event={"ID":"6c09fd91-63a7-4699-a611-61e6e7038a5e","Type":"ContainerDied","Data":"7e142f98e59be4db5e5bf4f217bd9b77ca05c57ab8d6c61bb2abeb4fd18c160d"} Oct 31 19:38:26.660832 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:26.660704 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e142f98e59be4db5e5bf4f217bd9b77ca05c57ab8d6c61bb2abeb4fd18c160d" Oct 31 19:38:27.456210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:27.443458 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:38:27.456210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:27.444587 5688 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:38:27.456210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:27.444637 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:38:28.437505 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:28.437470 5688 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:38:28.438090 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:28.438058 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:38:29.453208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:29.443480 5688 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:38:29.453208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:29.443535 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:38:30.457208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:30.449477 5688 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:38:30.457208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:30.449548 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:38:31.438473 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:31.438438 5688 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:38:31.438754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:31.438726 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:38:32.439636 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:32.439603 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:38:32.440111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:32.440089 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:38:32.440878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:32.440857 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:38:33.798570 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.798541 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb"] Oct 31 19:38:33.799011 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.798996 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6c09fd91-63a7-4699-a611-61e6e7038a5e" containerName="revoke-keys" Oct 31 19:38:33.799055 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.799013 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c09fd91-63a7-4699-a611-61e6e7038a5e" containerName="revoke-keys" Oct 31 19:38:33.799112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.799103 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c09fd91-63a7-4699-a611-61e6e7038a5e" containerName="revoke-keys" Oct 31 19:38:33.802377 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.802359 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.815658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.815633 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb"] Oct 31 19:38:33.854799 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.854773 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:33.855037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.855015 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:38:33.858439 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.858421 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.858526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.858458 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.858645 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.858626 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5h4\" (UniqueName: \"kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.959508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.959480 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5h4\" (UniqueName: \"kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.959627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.959554 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.959627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.959598 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.960102 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.960078 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:33.962000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:33.961975 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:34.018028 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:34.018002 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5h4\" (UniqueName: \"kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4\") pod \"maas-api-revoke-sub-798d20cc-685jb\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:34.117696 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:34.117641 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:34.492552 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:34.492530 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb"] Oct 31 19:38:34.495100 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:38:34.495074 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8779577_f7a0_428c_9ec2_a0e572f04c9c.slice/crio-64c5021c38675bcb1444db9dc7cd82c1a214db0d9aad5cb1a09919fd73c4c56b WatchSource:0}: Error finding container 64c5021c38675bcb1444db9dc7cd82c1a214db0d9aad5cb1a09919fd73c4c56b: Status 404 returned error can't find the container with id 64c5021c38675bcb1444db9dc7cd82c1a214db0d9aad5cb1a09919fd73c4c56b Oct 31 19:38:34.718961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:34.718917 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" event={"ID":"e8779577-f7a0-428c-9ec2-a0e572f04c9c","Type":"ContainerStarted","Data":"64c5021c38675bcb1444db9dc7cd82c1a214db0d9aad5cb1a09919fd73c4c56b"} Oct 31 19:38:35.725795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:35.725721 5688 generic.go:356] "Generic (PLEG): container finished" podID="e8779577-f7a0-428c-9ec2-a0e572f04c9c" containerID="e2c0bb36fb3fbf09a9fe41099ce8c15a80f1ec7e724fa3e38918bf18a8b2fdaf" exitCode=0 Oct 31 19:38:35.726146 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:35.725804 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" event={"ID":"e8779577-f7a0-428c-9ec2-a0e572f04c9c","Type":"ContainerDied","Data":"e2c0bb36fb3fbf09a9fe41099ce8c15a80f1ec7e724fa3e38918bf18a8b2fdaf"} Oct 31 19:38:36.876365 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.876342 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:36.989417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.989392 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4\" (UniqueName: \"kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4\") pod \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " Oct 31 19:38:36.989558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.989464 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca\") pod \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " Oct 31 19:38:36.989558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.989550 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca\") pod \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\" (UID: \"e8779577-f7a0-428c-9ec2-a0e572f04c9c\") " Oct 31 19:38:36.989884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.989859 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca" pod "e8779577-f7a0-428c-9ec2-a0e572f04c9c" (UID: "e8779577-f7a0-428c-9ec2-a0e572f04c9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:36.989986 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.989954 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:36.991590 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.991566 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4" pod "e8779577-f7a0-428c-9ec2-a0e572f04c9c" (UID: "e8779577-f7a0-428c-9ec2-a0e572f04c9c"). InnerVolumeSpecName "kube-api-access-hz5h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:36.991685 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:36.991607 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca" pod "e8779577-f7a0-428c-9ec2-a0e572f04c9c" (UID: "e8779577-f7a0-428c-9ec2-a0e572f04c9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:38:37.091258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:37.091229 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8779577-f7a0-428c-9ec2-a0e572f04c9c-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:37.091258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:37.091256 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5h4\" (UniqueName: \"kubernetes.io/projected/e8779577-f7a0-428c-9ec2-a0e572f04c9c-kube-api-access-hz5h4\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:37.738441 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:37.738410 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" Oct 31 19:38:37.738441 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:37.738419 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb" event={"ID":"e8779577-f7a0-428c-9ec2-a0e572f04c9c","Type":"ContainerDied","Data":"64c5021c38675bcb1444db9dc7cd82c1a214db0d9aad5cb1a09919fd73c4c56b"} Oct 31 19:38:37.738677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:37.738462 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64c5021c38675bcb1444db9dc7cd82c1a214db0d9aad5cb1a09919fd73c4c56b" Oct 31 19:38:45.386502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.386466 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7"] Oct 31 19:38:45.386938 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.386924 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e8779577-f7a0-428c-9ec2-a0e572f04c9c" containerName="revoke-keys" Oct 31 19:38:45.386985 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.386938 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8779577-f7a0-428c-9ec2-a0e572f04c9c" containerName="revoke-keys" Oct 31 19:38:45.387024 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.387015 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8779577-f7a0-428c-9ec2-a0e572f04c9c" containerName="revoke-keys" Oct 31 19:38:45.389979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.389952 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.484020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.483991 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.484020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.484030 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnk2\" (UniqueName: \"kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.484284 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.484169 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.495366 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.495345 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:45.495479 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.495405 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:38:45.497553 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.497532 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7"] Oct 31 19:38:45.585339 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.585318 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.585464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.585351 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnk2\" (UniqueName: \"kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.585464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.585390 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.585942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.585926 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.587837 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.587816 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.642042 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.641986 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnk2\" (UniqueName: \"kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2\") pod \"maas-api-revoke-sub-cd201420-ttcb7\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:45.702820 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:45.702799 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:46.077487 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:46.077459 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7"] Oct 31 19:38:46.079499 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:38:46.079466 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482b3eaa_1b55_4d3d_b315_9c0b5a5234ca.slice/crio-380e0c802dd082953c8cb645380ca3d6815825b7871311dbbec5270ce3ee1986 WatchSource:0}: Error finding container 380e0c802dd082953c8cb645380ca3d6815825b7871311dbbec5270ce3ee1986: Status 404 returned error can't find the container with id 380e0c802dd082953c8cb645380ca3d6815825b7871311dbbec5270ce3ee1986 Oct 31 19:38:46.788397 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:46.788349 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" event={"ID":"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca","Type":"ContainerStarted","Data":"a81ef0c4227b2e5d82b6abead8eb3f7d13ea70c40f423beb78711a163cf6fca0"} Oct 31 19:38:46.788799 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:46.788412 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" event={"ID":"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca","Type":"ContainerStarted","Data":"380e0c802dd082953c8cb645380ca3d6815825b7871311dbbec5270ce3ee1986"} Oct 31 19:38:46.905723 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:46.905667 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" podStartSLOduration=1.905653947 podStartE2EDuration="1.905653947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:46.903103053 +0000 UTC m=+4647.505244197" watchObservedRunningTime="2026-10-31 19:38:46.905653947 +0000 UTC m=+4647.507795090" Oct 31 19:38:47.800901 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:47.800862 5688 generic.go:356] "Generic (PLEG): container finished" podID="482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" containerID="a81ef0c4227b2e5d82b6abead8eb3f7d13ea70c40f423beb78711a163cf6fca0" exitCode=0 Oct 31 19:38:47.801634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:47.801605 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" event={"ID":"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca","Type":"ContainerDied","Data":"a81ef0c4227b2e5d82b6abead8eb3f7d13ea70c40f423beb78711a163cf6fca0"} Oct 31 19:38:48.949151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:48.949129 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:49.032484 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.032459 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca\") pod \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " Oct 31 19:38:49.032604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.032505 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca\") pod \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " Oct 31 19:38:49.032665 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.032603 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2\" (UniqueName: \"kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2\") pod \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\" (UID: \"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca\") " Oct 31 19:38:49.032862 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.032841 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca" pod "482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" (UID: "482b3eaa-1b55-4d3d-b315-9c0b5a5234ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:49.033114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.033096 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:49.034778 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.034751 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca" pod "482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" (UID: "482b3eaa-1b55-4d3d-b315-9c0b5a5234ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:38:49.034867 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.034769 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2" pod "482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" (UID: "482b3eaa-1b55-4d3d-b315-9c0b5a5234ca"). InnerVolumeSpecName "kube-api-access-zfnk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:49.134213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.134124 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:49.134213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.134155 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnk2\" (UniqueName: \"kubernetes.io/projected/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca-kube-api-access-zfnk2\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:38:49.814027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.813996 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" event={"ID":"482b3eaa-1b55-4d3d-b315-9c0b5a5234ca","Type":"ContainerDied","Data":"380e0c802dd082953c8cb645380ca3d6815825b7871311dbbec5270ce3ee1986"} Oct 31 19:38:49.814210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.814044 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380e0c802dd082953c8cb645380ca3d6815825b7871311dbbec5270ce3ee1986" Oct 31 19:38:49.814210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:49.814009 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7" Oct 31 19:38:54.904320 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:38:54.904284 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:38:56.313912 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.313880 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt"] Oct 31 19:38:56.314532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.314517 5688 cpu_manager.go:414] "containerMap: removing container" podUID="482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" containerName="revoke-keys" Oct 31 19:38:56.314577 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.314535 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" containerName="revoke-keys" Oct 31 19:38:56.314703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.314693 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" containerName="revoke-keys" Oct 31 19:38:56.317958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.317940 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.374559 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.374534 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt"] Oct 31 19:38:56.402456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.402428 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.402598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.402495 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.402664 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.402641 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvwb\" (UniqueName: \"kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.421265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.421234 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:56.421384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.421334 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:38:56.503896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.503867 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.504057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.503916 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.504057 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.503949 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvwb\" (UniqueName: \"kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.504547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.504522 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.506340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.506321 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.563774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.563744 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvwb\" (UniqueName: \"kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb\") pod \"maas-api-revoke-sub-bede1cbc-7xmzt\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:56.631391 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:56.631338 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:38:57.004990 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:57.004966 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt"] Oct 31 19:38:57.007127 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:38:57.007103 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec98734_9195_4438_ac02_cf6631535085.slice/crio-426dab8ff6187eef5dceadb71812c31089737ca10463ae1a6c44089032dd5763 WatchSource:0}: Error finding container 426dab8ff6187eef5dceadb71812c31089737ca10463ae1a6c44089032dd5763: Status 404 returned error can't find the container with id 426dab8ff6187eef5dceadb71812c31089737ca10463ae1a6c44089032dd5763 Oct 31 19:38:57.860361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:57.860322 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" event={"ID":"4ec98734-9195-4438-ac02-cf6631535085","Type":"ContainerStarted","Data":"bd0812a86cc7ad68388aaa72ddfc03ce87588308dc1cbcd1668e06b322d20598"} Oct 31 19:38:57.860776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:57.860379 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" event={"ID":"4ec98734-9195-4438-ac02-cf6631535085","Type":"ContainerStarted","Data":"426dab8ff6187eef5dceadb71812c31089737ca10463ae1a6c44089032dd5763"} Oct 31 19:38:57.977417 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:57.977369 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" podStartSLOduration=1.97735451 podStartE2EDuration="1.97735451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:57.974362071 +0000 UTC m=+4658.576503215" watchObservedRunningTime="2026-10-31 19:38:57.97735451 +0000 UTC m=+4658.579495654" Oct 31 19:38:58.866788 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:58.866756 5688 generic.go:356] "Generic (PLEG): container finished" podID="4ec98734-9195-4438-ac02-cf6631535085" containerID="bd0812a86cc7ad68388aaa72ddfc03ce87588308dc1cbcd1668e06b322d20598" exitCode=0 Oct 31 19:38:58.867225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:38:58.866800 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" event={"ID":"4ec98734-9195-4438-ac02-cf6631535085","Type":"ContainerDied","Data":"bd0812a86cc7ad68388aaa72ddfc03ce87588308dc1cbcd1668e06b322d20598"} Oct 31 19:39:00.016018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.015992 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:39:00.140752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.140681 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca\") pod \"4ec98734-9195-4438-ac02-cf6631535085\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " Oct 31 19:39:00.140752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.140718 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca\") pod \"4ec98734-9195-4438-ac02-cf6631535085\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " Oct 31 19:39:00.140752 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.140736 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb\" (UniqueName: \"kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb\") pod \"4ec98734-9195-4438-ac02-cf6631535085\" (UID: \"4ec98734-9195-4438-ac02-cf6631535085\") " Oct 31 19:39:00.141131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.141077 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca" pod "4ec98734-9195-4438-ac02-cf6631535085" (UID: "4ec98734-9195-4438-ac02-cf6631535085"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:00.141349 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.141330 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec98734-9195-4438-ac02-cf6631535085-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:00.143235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.143170 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb" pod "4ec98734-9195-4438-ac02-cf6631535085" (UID: "4ec98734-9195-4438-ac02-cf6631535085"). InnerVolumeSpecName "kube-api-access-fzvwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:00.143354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.143274 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca" pod "4ec98734-9195-4438-ac02-cf6631535085" (UID: "4ec98734-9195-4438-ac02-cf6631535085"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:00.242747 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.242720 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec98734-9195-4438-ac02-cf6631535085-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:00.242747 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.242746 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvwb\" (UniqueName: \"kubernetes.io/projected/4ec98734-9195-4438-ac02-cf6631535085-kube-api-access-fzvwb\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:00.879813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.879784 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" Oct 31 19:39:00.879979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.879817 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt" event={"ID":"4ec98734-9195-4438-ac02-cf6631535085","Type":"ContainerDied","Data":"426dab8ff6187eef5dceadb71812c31089737ca10463ae1a6c44089032dd5763"} Oct 31 19:39:00.879979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:00.879866 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426dab8ff6187eef5dceadb71812c31089737ca10463ae1a6c44089032dd5763" Oct 31 19:39:07.597931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.597896 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd"] Oct 31 19:39:07.598430 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.598088 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" podUID="ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" containerName="maas-api" containerID="cri-o://7b5f68936766e18efb1bc6af66470931c0528ebd0fdfef3dd94940721bacfc61" gracePeriod=30 Oct 31 19:39:07.922153 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.922118 5688 generic.go:356] "Generic (PLEG): container finished" podID="ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" containerID="7b5f68936766e18efb1bc6af66470931c0528ebd0fdfef3dd94940721bacfc61" exitCode=0 Oct 31 19:39:07.922333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.922218 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" event={"ID":"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e","Type":"ContainerDied","Data":"7b5f68936766e18efb1bc6af66470931c0528ebd0fdfef3dd94940721bacfc61"} Oct 31 19:39:07.922333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.922274 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" event={"ID":"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e","Type":"ContainerDied","Data":"c7e1e43d939a8bbbfc58c5ab8bb8e6fda2bf48913b956406f7771ea5535a453b"} Oct 31 19:39:07.922333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.922297 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e1e43d939a8bbbfc58c5ab8bb8e6fda2bf48913b956406f7771ea5535a453b" Oct 31 19:39:07.937072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:07.937054 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:39:08.032682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.031618 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\") pod \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " Oct 31 19:39:08.032682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.031774 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca\") pod \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " Oct 31 19:39:08.032682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.031808 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9\" (UniqueName: \"kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9\") pod \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " Oct 31 19:39:08.032682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.031853 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs\") pod \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\" (UID: \"ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e\") " Oct 31 19:39:08.034006 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.033971 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca" pod "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" (UID: "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:08.046495 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.046443 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls" pod "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" (UID: "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:08.048070 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.048026 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs" pod "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" (UID: "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:08.058205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.058021 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9" pod "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" (UID: "ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e"). InnerVolumeSpecName "kube-api-access-fcdb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:08.133628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.133592 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:08.133628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.133620 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:08.133628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.133629 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdb9\" (UniqueName: \"kubernetes.io/projected/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-kube-api-access-fcdb9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:08.133916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.133642 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:08.927471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:08.927396 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd" Oct 31 19:39:09.091886 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:09.091858 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd"] Oct 31 19:39:09.141815 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:09.141775 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4731a-76878445cf-2ljsd"] Oct 31 19:39:09.565606 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:09.565568 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" path="/var/lib/kubelet/pods/ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e/volumes" Oct 31 19:39:13.829794 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.829758 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc"] Oct 31 19:39:13.830268 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.830252 5688 cpu_manager.go:414] "containerMap: removing container" podUID="ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" containerName="maas-api" Oct 31 19:39:13.830332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.830270 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" containerName="maas-api" Oct 31 19:39:13.830332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.830282 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4ec98734-9195-4438-ac02-cf6631535085" containerName="revoke-keys" Oct 31 19:39:13.830332 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.830287 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec98734-9195-4438-ac02-cf6631535085" containerName="revoke-keys" Oct 31 19:39:13.830470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.830387 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec98734-9195-4438-ac02-cf6631535085" containerName="revoke-keys" Oct 31 19:39:13.830470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.830403 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac03bcd2-2b9a-4a65-9ab2-cf9ec22fd19e" containerName="maas-api" Oct 31 19:39:13.834967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.834942 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:13.903970 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.903932 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:13.904112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.904086 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:13.904200 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.904117 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chp8v\" (UniqueName: \"kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:13.913807 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.913788 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:39:13.913919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.913837 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:39:13.915833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:13.915814 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc"] Oct 31 19:39:14.005470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.005444 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.005626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.005530 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.005626 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.005554 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chp8v\" (UniqueName: \"kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.006271 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.006246 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.007925 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.007904 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.062467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.062444 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chp8v\" (UniqueName: \"kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v\") pod \"maas-api-revoke-sub-fd3b325f-5xbmc\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.150878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.150814 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:14.524567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.524532 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc"] Oct 31 19:39:14.531381 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:39:14.531352 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fcb598_2d84_4b86_aa01_e32958043126.slice/crio-656cd93525c5c4d5f979dfeb14798856835f08b946224a0322035d7c5e29d0d9 WatchSource:0}: Error finding container 656cd93525c5c4d5f979dfeb14798856835f08b946224a0322035d7c5e29d0d9: Status 404 returned error can't find the container with id 656cd93525c5c4d5f979dfeb14798856835f08b946224a0322035d7c5e29d0d9 Oct 31 19:39:14.879119 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.879086 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr"] Oct 31 19:39:14.884414 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.884398 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:14.915797 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.915773 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:14.915913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.915803 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgk8h\" (UniqueName: \"kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h\") pod \"payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:14.942581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.940491 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-0f0e4e-dockercfg-bghjm\"" Oct 31 19:39:14.942581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.940572 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-0f0e4e\"" Oct 31 19:39:14.964243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:14.943707 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr"] Oct 31 19:39:15.007110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.007074 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" event={"ID":"79fcb598-2d84-4b86-aa01-e32958043126","Type":"ContainerStarted","Data":"656cd93525c5c4d5f979dfeb14798856835f08b946224a0322035d7c5e29d0d9"} Oct 31 19:39:15.025359 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.025323 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:15.025552 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.025370 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgk8h\" (UniqueName: \"kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h\") pod \"payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:15.026035 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.026007 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:15.038592 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.038560 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb"] Oct 31 19:39:15.043114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.043087 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.083823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.083789 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgk8h\" (UniqueName: \"kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h\") pod \"payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:15.117879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.117851 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb"] Oct 31 19:39:15.126782 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.126757 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.126927 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.126791 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km22l\" (UniqueName: \"kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l\") pod \"payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.197954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.197890 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:15.227976 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.227941 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.228139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.228001 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km22l\" (UniqueName: \"kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l\") pod \"payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.228856 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.228830 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.295233 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.291535 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km22l\" (UniqueName: \"kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l\") pod \"payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.360744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.360713 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:15.620560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.620538 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr"] Oct 31 19:39:15.622808 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:39:15.622777 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021b5d79_468d_4a37_a477_f358b903f7ab.slice/crio-d19ae46b5307ebd73b7ee280feaeae82977f83c87cc0be974fcb4ab0cb52d74e WatchSource:0}: Error finding container d19ae46b5307ebd73b7ee280feaeae82977f83c87cc0be974fcb4ab0cb52d74e: Status 404 returned error can't find the container with id d19ae46b5307ebd73b7ee280feaeae82977f83c87cc0be974fcb4ab0cb52d74e Oct 31 19:39:15.773303 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:15.773276 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb"] Oct 31 19:39:15.775039 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:39:15.775017 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ded44c_70e0_403b_878b_ed91a81a3f4e.slice/crio-07074a3cd3a809d0c146504188318237c8a99379902b732202f1ecc004f560e5 WatchSource:0}: Error finding container 07074a3cd3a809d0c146504188318237c8a99379902b732202f1ecc004f560e5: Status 404 returned error can't find the container with id 07074a3cd3a809d0c146504188318237c8a99379902b732202f1ecc004f560e5 Oct 31 19:39:16.016448 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.016414 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" event={"ID":"04ded44c-70e0-403b-878b-ed91a81a3f4e","Type":"ContainerStarted","Data":"78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab"} Oct 31 19:39:16.016919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.016467 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" event={"ID":"04ded44c-70e0-403b-878b-ed91a81a3f4e","Type":"ContainerStarted","Data":"07074a3cd3a809d0c146504188318237c8a99379902b732202f1ecc004f560e5"} Oct 31 19:39:16.016919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.016506 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:16.020040 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.019922 5688 generic.go:356] "Generic (PLEG): container finished" podID="79fcb598-2d84-4b86-aa01-e32958043126" containerID="9d2e941c31d9ee3499dfb5c74251f278cd126eb2b6af37f156420fbc27cf5f8c" exitCode=0 Oct 31 19:39:16.020040 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.019989 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" event={"ID":"79fcb598-2d84-4b86-aa01-e32958043126","Type":"ContainerDied","Data":"9d2e941c31d9ee3499dfb5c74251f278cd126eb2b6af37f156420fbc27cf5f8c"} Oct 31 19:39:16.021798 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.021773 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" event={"ID":"021b5d79-468d-4a37-a477-f358b903f7ab","Type":"ContainerStarted","Data":"e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2"} Oct 31 19:39:16.021921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.021817 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" event={"ID":"021b5d79-468d-4a37-a477-f358b903f7ab","Type":"ContainerStarted","Data":"d19ae46b5307ebd73b7ee280feaeae82977f83c87cc0be974fcb4ab0cb52d74e"} Oct 31 19:39:16.021921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.021873 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:16.132881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.132820 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" podStartSLOduration=2.132806245 podStartE2EDuration="2.132806245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:16.130430669 +0000 UTC m=+4676.732571812" watchObservedRunningTime="2026-10-31 19:39:16.132806245 +0000 UTC m=+4676.734947444" Oct 31 19:39:16.380273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:16.376848 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" podStartSLOduration=2.376828319 podStartE2EDuration="2.376828319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:16.255582004 +0000 UTC m=+4676.857723139" watchObservedRunningTime="2026-10-31 19:39:16.376828319 +0000 UTC m=+4676.978969465" Oct 31 19:39:17.183760 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.183736 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:17.252396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.252366 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca\") pod \"79fcb598-2d84-4b86-aa01-e32958043126\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " Oct 31 19:39:17.252559 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.252440 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v\" (UniqueName: \"kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v\") pod \"79fcb598-2d84-4b86-aa01-e32958043126\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " Oct 31 19:39:17.252559 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.252475 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca\") pod \"79fcb598-2d84-4b86-aa01-e32958043126\" (UID: \"79fcb598-2d84-4b86-aa01-e32958043126\") " Oct 31 19:39:17.252697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.252676 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca" pod "79fcb598-2d84-4b86-aa01-e32958043126" (UID: "79fcb598-2d84-4b86-aa01-e32958043126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:17.253152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.253036 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:17.254843 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.254813 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v" pod "79fcb598-2d84-4b86-aa01-e32958043126" (UID: "79fcb598-2d84-4b86-aa01-e32958043126"). InnerVolumeSpecName "kube-api-access-chp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:17.254979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.254960 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca" pod "79fcb598-2d84-4b86-aa01-e32958043126" (UID: "79fcb598-2d84-4b86-aa01-e32958043126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:17.354325 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.354262 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chp8v\" (UniqueName: \"kubernetes.io/projected/79fcb598-2d84-4b86-aa01-e32958043126-kube-api-access-chp8v\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:17.354325 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:17.354287 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fcb598-2d84-4b86-aa01-e32958043126-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:18.034855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:18.034819 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" Oct 31 19:39:18.035039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:18.034852 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc" event={"ID":"79fcb598-2d84-4b86-aa01-e32958043126","Type":"ContainerDied","Data":"656cd93525c5c4d5f979dfeb14798856835f08b946224a0322035d7c5e29d0d9"} Oct 31 19:39:18.035039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:18.034909 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="656cd93525c5c4d5f979dfeb14798856835f08b946224a0322035d7c5e29d0d9" Oct 31 19:39:18.155552 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:18.155518 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87"] Oct 31 19:39:18.160542 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:18.160493 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22"] Oct 31 19:39:21.854907 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.854875 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg"] Oct 31 19:39:21.855420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.855405 5688 cpu_manager.go:414] "containerMap: removing container" podUID="79fcb598-2d84-4b86-aa01-e32958043126" containerName="revoke-keys" Oct 31 19:39:21.855465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.855421 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fcb598-2d84-4b86-aa01-e32958043126" containerName="revoke-keys" Oct 31 19:39:21.855521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.855513 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fcb598-2d84-4b86-aa01-e32958043126" containerName="revoke-keys" Oct 31 19:39:21.858917 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.858899 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:21.902742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.902710 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:21.902881 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.902747 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:21.902950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.902898 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7c5\" (UniqueName: \"kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:21.913290 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.913270 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:39:21.913589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.913566 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:39:21.916261 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.916237 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg"] Oct 31 19:39:21.924449 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.924425 5688 scope.go:122] "RemoveContainer" containerID="7b5f68936766e18efb1bc6af66470931c0528ebd0fdfef3dd94940721bacfc61" Oct 31 19:39:21.936643 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:21.936622 5688 scope.go:122] "RemoveContainer" containerID="d248e6106601c4b25b4f4fc404995af598506881585bf250968afadc523f9bec" Oct 31 19:39:22.003968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.003945 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.004088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.003978 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.004088 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.004032 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7c5\" (UniqueName: \"kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.004634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.004611 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.006373 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.006353 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.061568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.061546 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7c5\" (UniqueName: \"kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5\") pod \"maas-api-revoke-sub-89b9ca8b-zgdtg\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.172015 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.171951 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:22.556866 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:22.556837 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg"] Oct 31 19:39:22.558138 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:39:22.558108 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166aa54d_69fa_4bdb_b85d_72bfc894d235.slice/crio-c5ca351e3c77e2b282ab3e3cf961053994439bfb74ea756a20d435d21dcfe66c WatchSource:0}: Error finding container c5ca351e3c77e2b282ab3e3cf961053994439bfb74ea756a20d435d21dcfe66c: Status 404 returned error can't find the container with id c5ca351e3c77e2b282ab3e3cf961053994439bfb74ea756a20d435d21dcfe66c Oct 31 19:39:23.065091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:23.065056 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" event={"ID":"166aa54d-69fa-4bdb-b85d-72bfc894d235","Type":"ContainerStarted","Data":"697c9b3fa8153fb5ed0b28522b65dfbb068f204469f23da302fc5ccb582bfb0f"} Oct 31 19:39:23.065447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:23.065114 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" event={"ID":"166aa54d-69fa-4bdb-b85d-72bfc894d235","Type":"ContainerStarted","Data":"c5ca351e3c77e2b282ab3e3cf961053994439bfb74ea756a20d435d21dcfe66c"} Oct 31 19:39:23.189511 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:23.189417 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" podStartSLOduration=2.189405495 podStartE2EDuration="2.189405495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:23.187776774 +0000 UTC m=+4683.789917918" watchObservedRunningTime="2026-10-31 19:39:23.189405495 +0000 UTC m=+4683.791546655" Oct 31 19:39:24.072757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:24.072719 5688 generic.go:356] "Generic (PLEG): container finished" podID="166aa54d-69fa-4bdb-b85d-72bfc894d235" containerID="697c9b3fa8153fb5ed0b28522b65dfbb068f204469f23da302fc5ccb582bfb0f" exitCode=0 Oct 31 19:39:24.073133 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:24.072807 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" event={"ID":"166aa54d-69fa-4bdb-b85d-72bfc894d235","Type":"ContainerDied","Data":"697c9b3fa8153fb5ed0b28522b65dfbb068f204469f23da302fc5ccb582bfb0f"} Oct 31 19:39:24.904349 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:39:24.904316 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:39:25.226197 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.226164 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:25.334504 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.334482 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca\") pod \"166aa54d-69fa-4bdb-b85d-72bfc894d235\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " Oct 31 19:39:25.334633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.334522 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca\") pod \"166aa54d-69fa-4bdb-b85d-72bfc894d235\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " Oct 31 19:39:25.334676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.334639 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5\" (UniqueName: \"kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5\") pod \"166aa54d-69fa-4bdb-b85d-72bfc894d235\" (UID: \"166aa54d-69fa-4bdb-b85d-72bfc894d235\") " Oct 31 19:39:25.334802 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.334782 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca" pod "166aa54d-69fa-4bdb-b85d-72bfc894d235" (UID: "166aa54d-69fa-4bdb-b85d-72bfc894d235"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:25.334961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.334943 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:25.336600 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.336580 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca" pod "166aa54d-69fa-4bdb-b85d-72bfc894d235" (UID: "166aa54d-69fa-4bdb-b85d-72bfc894d235"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:25.336703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.336684 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5" pod "166aa54d-69fa-4bdb-b85d-72bfc894d235" (UID: "166aa54d-69fa-4bdb-b85d-72bfc894d235"). InnerVolumeSpecName "kube-api-access-sl7c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:25.436614 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.436583 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166aa54d-69fa-4bdb-b85d-72bfc894d235-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:25.436740 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:25.436618 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl7c5\" (UniqueName: \"kubernetes.io/projected/166aa54d-69fa-4bdb-b85d-72bfc894d235-kube-api-access-sl7c5\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:26.084056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:26.084029 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" Oct 31 19:39:26.084282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:26.084066 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg" event={"ID":"166aa54d-69fa-4bdb-b85d-72bfc894d235","Type":"ContainerDied","Data":"c5ca351e3c77e2b282ab3e3cf961053994439bfb74ea756a20d435d21dcfe66c"} Oct 31 19:39:26.084282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:26.084114 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5ca351e3c77e2b282ab3e3cf961053994439bfb74ea756a20d435d21dcfe66c" Oct 31 19:39:27.032345 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:27.032313 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:39:27.032769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:27.032390 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:39:30.073415 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.073347 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr"] Oct 31 19:39:30.073851 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.073827 5688 cpu_manager.go:414] "containerMap: removing container" podUID="166aa54d-69fa-4bdb-b85d-72bfc894d235" containerName="revoke-keys" Oct 31 19:39:30.073851 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.073837 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166aa54d-69fa-4bdb-b85d-72bfc894d235" containerName="revoke-keys" Oct 31 19:39:30.073931 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.073925 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="166aa54d-69fa-4bdb-b85d-72bfc894d235" containerName="revoke-keys" Oct 31 19:39:30.077025 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.077009 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.180225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.180197 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:39:30.180225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.180217 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:39:30.182652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.182631 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr"] Oct 31 19:39:30.190616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.190597 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xj7h\" (UniqueName: \"kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.190702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.190645 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.190702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.190663 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.291866 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.291843 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xj7h\" (UniqueName: \"kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.292012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.291889 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.292012 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.291905 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.292581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.292563 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.294325 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.294307 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.348554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.348494 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xj7h\" (UniqueName: \"kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h\") pod \"maas-api-revoke-sub-254a4e70-5gzxr\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.389734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.389710 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:30.763771 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:30.763748 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr"] Oct 31 19:39:30.766058 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:39:30.766010 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af41b22_6920_4509_8d1d_9eaba237d680.slice/crio-70f8b6719174f498b10eddec33964d6c77f0a88adb5b275121e184bfd5773abd WatchSource:0}: Error finding container 70f8b6719174f498b10eddec33964d6c77f0a88adb5b275121e184bfd5773abd: Status 404 returned error can't find the container with id 70f8b6719174f498b10eddec33964d6c77f0a88adb5b275121e184bfd5773abd Oct 31 19:39:31.112779 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:31.112702 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" event={"ID":"4af41b22-6920-4509-8d1d-9eaba237d680","Type":"ContainerStarted","Data":"70f8b6719174f498b10eddec33964d6c77f0a88adb5b275121e184bfd5773abd"} Oct 31 19:39:32.120774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:32.120731 5688 generic.go:356] "Generic (PLEG): container finished" podID="4af41b22-6920-4509-8d1d-9eaba237d680" containerID="2f08f59faefff069754ff2a9b65926acdbc54206ca1433511818c39e0fa1b48a" exitCode=0 Oct 31 19:39:32.121161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:32.120868 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" event={"ID":"4af41b22-6920-4509-8d1d-9eaba237d680","Type":"ContainerDied","Data":"2f08f59faefff069754ff2a9b65926acdbc54206ca1433511818c39e0fa1b48a"} Oct 31 19:39:33.176807 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.176745 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" podUID="3c10670a-933d-4f20-a46d-cc4cb63e7091" containerName="main" containerID="cri-o://59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37" gracePeriod=45 Oct 31 19:39:33.181548 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.179985 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" containerID="cri-o://499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99" gracePeriod=45 Oct 31 19:39:33.342694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.342666 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:33.445256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.445229 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 19:39:33.447029 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.447008 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca\") pod \"4af41b22-6920-4509-8d1d-9eaba237d680\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " Oct 31 19:39:33.447174 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.447163 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h\" (UniqueName: \"kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h\") pod \"4af41b22-6920-4509-8d1d-9eaba237d680\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " Oct 31 19:39:33.447289 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.447215 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca\") pod \"4af41b22-6920-4509-8d1d-9eaba237d680\" (UID: \"4af41b22-6920-4509-8d1d-9eaba237d680\") " Oct 31 19:39:33.447379 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.447355 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca" pod "4af41b22-6920-4509-8d1d-9eaba237d680" (UID: "4af41b22-6920-4509-8d1d-9eaba237d680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:33.447648 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.447627 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.448826 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.448809 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 19:39:33.449238 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.449220 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h" pod "4af41b22-6920-4509-8d1d-9eaba237d680" (UID: "4af41b22-6920-4509-8d1d-9eaba237d680"). InnerVolumeSpecName "kube-api-access-6xj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:33.449336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.449316 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca" pod "4af41b22-6920-4509-8d1d-9eaba237d680" (UID: "4af41b22-6920-4509-8d1d-9eaba237d680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:33.548395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548367 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs\" (UniqueName: \"kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs\") pod \"603d0d27-08f1-4cf0-8979-462837af1f29\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " Oct 31 19:39:33.548569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548413 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home\") pod \"3c10670a-933d-4f20-a46d-cc4cb63e7091\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " Oct 31 19:39:33.548569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548462 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs\") pod \"3c10670a-933d-4f20-a46d-cc4cb63e7091\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " Oct 31 19:39:33.548569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548510 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home\") pod \"603d0d27-08f1-4cf0-8979-462837af1f29\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " Oct 31 19:39:33.548569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548539 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir\") pod \"3c10670a-933d-4f20-a46d-cc4cb63e7091\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548593 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir\") pod \"603d0d27-08f1-4cf0-8979-462837af1f29\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548628 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache\") pod \"3c10670a-933d-4f20-a46d-cc4cb63e7091\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548698 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm\") pod \"603d0d27-08f1-4cf0-8979-462837af1f29\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548727 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t\" (UniqueName: \"kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t\") pod \"603d0d27-08f1-4cf0-8979-462837af1f29\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548739 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home" pod "3c10670a-933d-4f20-a46d-cc4cb63e7091" (UID: "3c10670a-933d-4f20-a46d-cc4cb63e7091"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548778 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm\") pod \"3c10670a-933d-4f20-a46d-cc4cb63e7091\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548806 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache\") pod \"603d0d27-08f1-4cf0-8979-462837af1f29\" (UID: \"603d0d27-08f1-4cf0-8979-462837af1f29\") " Oct 31 19:39:33.548834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548827 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr\" (UniqueName: \"kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr\") pod \"3c10670a-933d-4f20-a46d-cc4cb63e7091\" (UID: \"3c10670a-933d-4f20-a46d-cc4cb63e7091\") " Oct 31 19:39:33.549175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548922 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir" pod "3c10670a-933d-4f20-a46d-cc4cb63e7091" (UID: "3c10670a-933d-4f20-a46d-cc4cb63e7091"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.549175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.548982 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache" pod "3c10670a-933d-4f20-a46d-cc4cb63e7091" (UID: "3c10670a-933d-4f20-a46d-cc4cb63e7091"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.549280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549222 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir" pod "603d0d27-08f1-4cf0-8979-462837af1f29" (UID: "603d0d27-08f1-4cf0-8979-462837af1f29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549607 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home" pod "603d0d27-08f1-4cf0-8979-462837af1f29" (UID: "603d0d27-08f1-4cf0-8979-462837af1f29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549627 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xj7h\" (UniqueName: \"kubernetes.io/projected/4af41b22-6920-4509-8d1d-9eaba237d680-kube-api-access-6xj7h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549649 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af41b22-6920-4509-8d1d-9eaba237d680-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549666 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549680 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549693 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549705 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.549823 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.549792 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache" pod "603d0d27-08f1-4cf0-8979-462837af1f29" (UID: "603d0d27-08f1-4cf0-8979-462837af1f29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.551520 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.551489 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs" pod "603d0d27-08f1-4cf0-8979-462837af1f29" (UID: "603d0d27-08f1-4cf0-8979-462837af1f29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:33.551649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.551607 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs" pod "3c10670a-933d-4f20-a46d-cc4cb63e7091" (UID: "3c10670a-933d-4f20-a46d-cc4cb63e7091"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:33.551720 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.551647 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr" pod "3c10670a-933d-4f20-a46d-cc4cb63e7091" (UID: "3c10670a-933d-4f20-a46d-cc4cb63e7091"). InnerVolumeSpecName "kube-api-access-5cjpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:33.551894 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.551863 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm" pod "603d0d27-08f1-4cf0-8979-462837af1f29" (UID: "603d0d27-08f1-4cf0-8979-462837af1f29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.552171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.552119 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t" pod "603d0d27-08f1-4cf0-8979-462837af1f29" (UID: "603d0d27-08f1-4cf0-8979-462837af1f29"). InnerVolumeSpecName "kube-api-access-js47t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:33.552609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.552573 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm" pod "3c10670a-933d-4f20-a46d-cc4cb63e7091" (UID: "3c10670a-933d-4f20-a46d-cc4cb63e7091"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:33.650965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.650938 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d0d27-08f1-4cf0-8979-462837af1f29-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.650965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.650964 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c10670a-933d-4f20-a46d-cc4cb63e7091-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.651151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.650972 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.651151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.650981 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.651151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.650990 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js47t\" (UniqueName: \"kubernetes.io/projected/603d0d27-08f1-4cf0-8979-462837af1f29-kube-api-access-js47t\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.651151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.650998 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c10670a-933d-4f20-a46d-cc4cb63e7091-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.651151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.651006 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/603d0d27-08f1-4cf0-8979-462837af1f29-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:33.651151 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:33.651016 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjpr\" (UniqueName: \"kubernetes.io/projected/3c10670a-933d-4f20-a46d-cc4cb63e7091-kube-api-access-5cjpr\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:39:34.133742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.133713 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" Oct 31 19:39:34.133742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.133725 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr" event={"ID":"4af41b22-6920-4509-8d1d-9eaba237d680","Type":"ContainerDied","Data":"70f8b6719174f498b10eddec33964d6c77f0a88adb5b275121e184bfd5773abd"} Oct 31 19:39:34.133957 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.133775 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f8b6719174f498b10eddec33964d6c77f0a88adb5b275121e184bfd5773abd" Oct 31 19:39:34.135109 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.135083 5688 generic.go:356] "Generic (PLEG): container finished" podID="603d0d27-08f1-4cf0-8979-462837af1f29" containerID="499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99" exitCode=0 Oct 31 19:39:34.135295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.135125 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" Oct 31 19:39:34.135295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.135164 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" event={"ID":"603d0d27-08f1-4cf0-8979-462837af1f29","Type":"ContainerDied","Data":"499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99"} Oct 31 19:39:34.135295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.135222 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22" event={"ID":"603d0d27-08f1-4cf0-8979-462837af1f29","Type":"ContainerDied","Data":"1e1951549e1fc7a295c5f99adcbd014fb17fa326348b26c74f05b37ec38d999f"} Oct 31 19:39:34.135295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.135244 5688 scope.go:122] "RemoveContainer" containerID="499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99" Oct 31 19:39:34.136879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.136694 5688 generic.go:356] "Generic (PLEG): container finished" podID="3c10670a-933d-4f20-a46d-cc4cb63e7091" containerID="59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37" exitCode=0 Oct 31 19:39:34.136879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.136755 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" event={"ID":"3c10670a-933d-4f20-a46d-cc4cb63e7091","Type":"ContainerDied","Data":"59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37"} Oct 31 19:39:34.136879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.136774 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" event={"ID":"3c10670a-933d-4f20-a46d-cc4cb63e7091","Type":"ContainerDied","Data":"02c1b1dfdb11c1350494eefad62dd26d2e7566ab44dac70647bd80185b586e2a"} Oct 31 19:39:34.136879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.136815 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87" Oct 31 19:39:34.144863 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.144845 5688 scope.go:122] "RemoveContainer" containerID="499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99" Oct 31 19:39:34.145208 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:39:34.145162 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99\": container with ID starting with 499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99 not found: ID does not exist" containerID="499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99" Oct 31 19:39:34.145331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.145209 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99"} err="failed to get container status \"499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99\": rpc error: code = NotFound desc = could not find container \"499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99\": container with ID starting with 499195af50cf52ff1b24e0ca3fdf27f63b2412f1e1c50b9dfe100f58bc439d99 not found: ID does not exist" Oct 31 19:39:34.145331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.145227 5688 scope.go:122] "RemoveContainer" containerID="59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37" Oct 31 19:39:34.153539 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.153519 5688 scope.go:122] "RemoveContainer" containerID="59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37" Oct 31 19:39:34.153813 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:39:34.153796 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37\": container with ID starting with 59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37 not found: ID does not exist" containerID="59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37" Oct 31 19:39:34.153885 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.153818 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37"} err="failed to get container status \"59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37\": rpc error: code = NotFound desc = could not find container \"59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37\": container with ID starting with 59d31693dc8504b3e8626037321da61f8174738cf5cea4a70155a27d48e74c37 not found: ID does not exist" Oct 31 19:39:34.356655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.356631 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22"] Oct 31 19:39:34.363895 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.363875 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/premium-w0-a8587d-kserve-f9fcc6555-whj22"] Oct 31 19:39:34.472669 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.472611 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87"] Oct 31 19:39:34.476897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:34.476872 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-w0-a8587d-kserve-5b8bc566b6-8rr87"] Oct 31 19:39:35.565850 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:35.565812 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c10670a-933d-4f20-a46d-cc4cb63e7091" path="/var/lib/kubelet/pods/3c10670a-933d-4f20-a46d-cc4cb63e7091/volumes" Oct 31 19:39:35.566443 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:39:35.566424 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" path="/var/lib/kubelet/pods/603d0d27-08f1-4cf0-8979-462837af1f29/volumes" Oct 31 19:39:54.906780 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:39:54.906739 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:40:04.349649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:04.349615 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k"] Oct 31 19:40:05.584806 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:05.584773 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w"] Oct 31 19:40:05.585347 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:05.584999 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" podUID="b3ad3539-217c-40c4-a417-17577eebe8d7" containerName="maas-api" containerID="cri-o://88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8" gracePeriod=30 Oct 31 19:40:06.849963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:06.849939 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:40:07.020066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.020041 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca\") pod \"b3ad3539-217c-40c4-a417-17577eebe8d7\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " Oct 31 19:40:07.020258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.020147 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs\") pod \"b3ad3539-217c-40c4-a417-17577eebe8d7\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " Oct 31 19:40:07.020258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.020168 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\") pod \"b3ad3539-217c-40c4-a417-17577eebe8d7\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " Oct 31 19:40:07.020258 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.020245 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb\" (UniqueName: \"kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb\") pod \"b3ad3539-217c-40c4-a417-17577eebe8d7\" (UID: \"b3ad3539-217c-40c4-a417-17577eebe8d7\") " Oct 31 19:40:07.020487 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.020460 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca" pod "b3ad3539-217c-40c4-a417-17577eebe8d7" (UID: "b3ad3539-217c-40c4-a417-17577eebe8d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:07.020633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.020611 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad3539-217c-40c4-a417-17577eebe8d7-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:07.022397 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.022367 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs" pod "b3ad3539-217c-40c4-a417-17577eebe8d7" (UID: "b3ad3539-217c-40c4-a417-17577eebe8d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:07.022508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.022437 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb" pod "b3ad3539-217c-40c4-a417-17577eebe8d7" (UID: "b3ad3539-217c-40c4-a417-17577eebe8d7"). InnerVolumeSpecName "kube-api-access-f4dtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:07.022508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.022478 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls" pod "b3ad3539-217c-40c4-a417-17577eebe8d7" (UID: "b3ad3539-217c-40c4-a417-17577eebe8d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:07.121833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.121800 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:07.121833 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.121832 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ad3539-217c-40c4-a417-17577eebe8d7-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:07.122045 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.121849 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4dtb\" (UniqueName: \"kubernetes.io/projected/b3ad3539-217c-40c4-a417-17577eebe8d7-kube-api-access-f4dtb\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:07.325079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.324991 5688 generic.go:356] "Generic (PLEG): container finished" podID="b3ad3539-217c-40c4-a417-17577eebe8d7" containerID="88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8" exitCode=0 Oct 31 19:40:07.325079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.325042 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" Oct 31 19:40:07.325079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.325065 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" event={"ID":"b3ad3539-217c-40c4-a417-17577eebe8d7","Type":"ContainerDied","Data":"88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8"} Oct 31 19:40:07.325358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.325115 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w" event={"ID":"b3ad3539-217c-40c4-a417-17577eebe8d7","Type":"ContainerDied","Data":"c803e2eb5680851d38ec95291032908406019c45dc37749f35b5d6d8503805d0"} Oct 31 19:40:07.325358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.325137 5688 scope.go:122] "RemoveContainer" containerID="88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8" Oct 31 19:40:07.335136 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.335121 5688 scope.go:122] "RemoveContainer" containerID="88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8" Oct 31 19:40:07.335398 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:40:07.335381 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8\": container with ID starting with 88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8 not found: ID does not exist" containerID="88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8" Oct 31 19:40:07.335456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.335405 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8"} err="failed to get container status \"88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8\": rpc error: code = NotFound desc = could not find container \"88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8\": container with ID starting with 88b97077677b625f52109a981ac492e3879dd983fb8cac2242ae1c3d30891bb8 not found: ID does not exist" Oct 31 19:40:07.491758 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.491725 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w"] Oct 31 19:40:07.495324 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.495298 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d1ff32-7fb975d567-5qf5w"] Oct 31 19:40:07.566400 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:07.566364 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ad3539-217c-40c4-a417-17577eebe8d7" path="/var/lib/kubelet/pods/b3ad3539-217c-40c4-a417-17577eebe8d7/volumes" Oct 31 19:40:09.350029 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:09.349986 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerName="payload-processing" containerID="cri-o://a1a432708b4bb4c08075040de11587406b63239bcb39609e82c4ee4c432c59ea" gracePeriod=25 Oct 31 19:40:15.371936 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:15.371897 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:40:24.904613 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:40:24.904581 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:40:25.371916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:25.371876 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:40:28.882497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.882461 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth"] Oct 31 19:40:28.882989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.882911 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3c10670a-933d-4f20-a46d-cc4cb63e7091" containerName="main" Oct 31 19:40:28.882989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.882926 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c10670a-933d-4f20-a46d-cc4cb63e7091" containerName="main" Oct 31 19:40:28.882989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.882951 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4af41b22-6920-4509-8d1d-9eaba237d680" containerName="revoke-keys" Oct 31 19:40:28.882989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.882959 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af41b22-6920-4509-8d1d-9eaba237d680" containerName="revoke-keys" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883003 5688 cpu_manager.go:414] "containerMap: removing container" podUID="b3ad3539-217c-40c4-a417-17577eebe8d7" containerName="maas-api" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883014 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ad3539-217c-40c4-a417-17577eebe8d7" containerName="maas-api" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883028 5688 cpu_manager.go:414] "containerMap: removing container" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883035 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883165 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ad3539-217c-40c4-a417-17577eebe8d7" containerName="maas-api" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883197 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="603d0d27-08f1-4cf0-8979-462837af1f29" containerName="main" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883214 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c10670a-933d-4f20-a46d-cc4cb63e7091" containerName="main" Oct 31 19:40:28.883240 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.883228 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af41b22-6920-4509-8d1d-9eaba237d680" containerName="revoke-keys" Oct 31 19:40:28.888208 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.888173 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:28.944494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.944461 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth"] Oct 31 19:40:28.972889 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.972859 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:28.973059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.972912 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhdn\" (UniqueName: \"kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:28.973118 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.973076 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:28.990819 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.990798 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:40:28.990949 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:28.990865 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:40:29.074419 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.074382 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.074609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.074447 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhdn\" (UniqueName: \"kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.074609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.074527 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.075177 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.075153 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.077410 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.077386 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.140394 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.140210 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhdn\" (UniqueName: \"kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn\") pod \"maas-api-revoke-sub-67a12e32-mllth\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.202558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.202533 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:29.488336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.488300 5688 generic.go:356] "Generic (PLEG): container finished" podID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerID="a1a432708b4bb4c08075040de11587406b63239bcb39609e82c4ee4c432c59ea" exitCode=0 Oct 31 19:40:29.488498 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.488372 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" event={"ID":"fca7b0d1-f4e1-464f-90c2-11723eed536f","Type":"ContainerDied","Data":"a1a432708b4bb4c08075040de11587406b63239bcb39609e82c4ee4c432c59ea"} Oct 31 19:40:29.588241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.588216 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:40:29.613902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.613883 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth"] Oct 31 19:40:29.615547 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:40:29.615524 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18309cf5_0f02_4e22_8abc_416b28542238.slice/crio-48def475e152be241025622c8434b9c5ce85a5ea2d74abc296f59cbfc2ca4ce4 WatchSource:0}: Error finding container 48def475e152be241025622c8434b9c5ce85a5ea2d74abc296f59cbfc2ca4ce4: Status 404 returned error can't find the container with id 48def475e152be241025622c8434b9c5ce85a5ea2d74abc296f59cbfc2ca4ce4 Oct 31 19:40:29.689592 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.689568 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j\" (UniqueName: \"kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j\") pod \"fca7b0d1-f4e1-464f-90c2-11723eed536f\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " Oct 31 19:40:29.689708 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.689616 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume\") pod \"fca7b0d1-f4e1-464f-90c2-11723eed536f\" (UID: \"fca7b0d1-f4e1-464f-90c2-11723eed536f\") " Oct 31 19:40:29.690009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.689972 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume" pod "fca7b0d1-f4e1-464f-90c2-11723eed536f" (UID: "fca7b0d1-f4e1-464f-90c2-11723eed536f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:29.691776 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.691757 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j" pod "fca7b0d1-f4e1-464f-90c2-11723eed536f" (UID: "fca7b0d1-f4e1-464f-90c2-11723eed536f"). InnerVolumeSpecName "kube-api-access-dtx2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:29.794941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.793172 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtx2j\" (UniqueName: \"kubernetes.io/projected/fca7b0d1-f4e1-464f-90c2-11723eed536f-kube-api-access-dtx2j\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:29.794941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:29.793227 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b0d1-f4e1-464f-90c2-11723eed536f-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:30.498202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.497953 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" Oct 31 19:40:30.498202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.497998 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k" event={"ID":"fca7b0d1-f4e1-464f-90c2-11723eed536f","Type":"ContainerDied","Data":"b79ff9b530dd81d911131340c6899e5e0ee6f3cd60832fffe850083b67928ef7"} Oct 31 19:40:30.498202 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.498063 5688 scope.go:122] "RemoveContainer" containerID="a1a432708b4bb4c08075040de11587406b63239bcb39609e82c4ee4c432c59ea" Oct 31 19:40:30.500306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.500278 5688 generic.go:356] "Generic (PLEG): container finished" podID="18309cf5-0f02-4e22-8abc-416b28542238" containerID="0eda36d3f7561d54a7f1ece1523571763401ac7d2972b443bbe9b9444e815ac8" exitCode=0 Oct 31 19:40:30.500420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.500351 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" event={"ID":"18309cf5-0f02-4e22-8abc-416b28542238","Type":"ContainerDied","Data":"0eda36d3f7561d54a7f1ece1523571763401ac7d2972b443bbe9b9444e815ac8"} Oct 31 19:40:30.500420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.500389 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" event={"ID":"18309cf5-0f02-4e22-8abc-416b28542238","Type":"ContainerStarted","Data":"48def475e152be241025622c8434b9c5ce85a5ea2d74abc296f59cbfc2ca4ce4"} Oct 31 19:40:30.667267 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.667237 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k"] Oct 31 19:40:30.717668 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:30.717605 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d1ff32-7f6ff4949-d854k"] Oct 31 19:40:31.569036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.569006 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" path="/var/lib/kubelet/pods/fca7b0d1-f4e1-464f-90c2-11723eed536f/volumes" Oct 31 19:40:31.636589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.636567 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:31.710673 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.710648 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca\") pod \"18309cf5-0f02-4e22-8abc-416b28542238\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " Oct 31 19:40:31.710825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.710700 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn\" (UniqueName: \"kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn\") pod \"18309cf5-0f02-4e22-8abc-416b28542238\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " Oct 31 19:40:31.710825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.710779 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca\") pod \"18309cf5-0f02-4e22-8abc-416b28542238\" (UID: \"18309cf5-0f02-4e22-8abc-416b28542238\") " Oct 31 19:40:31.711060 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.711034 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca" pod "18309cf5-0f02-4e22-8abc-416b28542238" (UID: "18309cf5-0f02-4e22-8abc-416b28542238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:31.711789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.711747 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:31.713382 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.713356 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn" pod "18309cf5-0f02-4e22-8abc-416b28542238" (UID: "18309cf5-0f02-4e22-8abc-416b28542238"). InnerVolumeSpecName "kube-api-access-jxhdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:31.713671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.713647 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca" pod "18309cf5-0f02-4e22-8abc-416b28542238" (UID: "18309cf5-0f02-4e22-8abc-416b28542238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:31.812707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.812624 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18309cf5-0f02-4e22-8abc-416b28542238-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:31.812707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:31.812652 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhdn\" (UniqueName: \"kubernetes.io/projected/18309cf5-0f02-4e22-8abc-416b28542238-kube-api-access-jxhdn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:32.514127 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:32.514096 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" Oct 31 19:40:32.514316 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:32.514097 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth" event={"ID":"18309cf5-0f02-4e22-8abc-416b28542238","Type":"ContainerDied","Data":"48def475e152be241025622c8434b9c5ce85a5ea2d74abc296f59cbfc2ca4ce4"} Oct 31 19:40:32.514316 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:32.514271 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48def475e152be241025622c8434b9c5ce85a5ea2d74abc296f59cbfc2ca4ce4" Oct 31 19:40:40.291398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.291359 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx"] Oct 31 19:40:40.292032 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.292010 5688 cpu_manager.go:414] "containerMap: removing container" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerName="payload-processing" Oct 31 19:40:40.292124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.292035 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerName="payload-processing" Oct 31 19:40:40.292124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.292068 5688 cpu_manager.go:414] "containerMap: removing container" podUID="18309cf5-0f02-4e22-8abc-416b28542238" containerName="revoke-keys" Oct 31 19:40:40.292124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.292078 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18309cf5-0f02-4e22-8abc-416b28542238" containerName="revoke-keys" Oct 31 19:40:40.292314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.292264 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca7b0d1-f4e1-464f-90c2-11723eed536f" containerName="payload-processing" Oct 31 19:40:40.292314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.292287 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="18309cf5-0f02-4e22-8abc-416b28542238" containerName="revoke-keys" Oct 31 19:40:40.295634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.295614 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.349167 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.349144 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:40:40.398952 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.398927 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:40:40.401290 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.401266 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx"] Oct 31 19:40:40.401538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.401515 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.401675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.401662 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.401729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.401697 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwt89\" (UniqueName: \"kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.502494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.502467 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.502646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.502561 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.502646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.502588 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwt89\" (UniqueName: \"kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.503170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.503137 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.505031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.505010 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.559915 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.559866 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwt89\" (UniqueName: \"kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89\") pod \"maas-api-revoke-sub-3833fc3a-4j6fx\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.607911 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.607887 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:40.983292 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:40.983263 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx"] Oct 31 19:40:40.987356 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:40:40.987332 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04df1b12_1787_4992_9563_104c12b5e2ac.slice/crio-31a92d5644cd16544c33a375f4509e6476f7b3a3a9f1c5aaba948a34876015f0 WatchSource:0}: Error finding container 31a92d5644cd16544c33a375f4509e6476f7b3a3a9f1c5aaba948a34876015f0: Status 404 returned error can't find the container with id 31a92d5644cd16544c33a375f4509e6476f7b3a3a9f1c5aaba948a34876015f0 Oct 31 19:40:41.572801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:41.572709 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" event={"ID":"04df1b12-1787-4992-9563-104c12b5e2ac","Type":"ContainerStarted","Data":"cd2d7d4e151a8d057bb3f4cd6132016a1c164a853cc91f25bfb6d4c92d7db588"} Oct 31 19:40:41.572801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:41.572764 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" event={"ID":"04df1b12-1787-4992-9563-104c12b5e2ac","Type":"ContainerStarted","Data":"31a92d5644cd16544c33a375f4509e6476f7b3a3a9f1c5aaba948a34876015f0"} Oct 31 19:40:41.691428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:41.691373 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" podStartSLOduration=1.691354999 podStartE2EDuration="1.691354999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:41.689102583 +0000 UTC m=+4762.291243730" watchObservedRunningTime="2026-10-31 19:40:41.691354999 +0000 UTC m=+4762.293496144" Oct 31 19:40:42.584112 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:42.584075 5688 generic.go:356] "Generic (PLEG): container finished" podID="04df1b12-1787-4992-9563-104c12b5e2ac" containerID="cd2d7d4e151a8d057bb3f4cd6132016a1c164a853cc91f25bfb6d4c92d7db588" exitCode=0 Oct 31 19:40:42.584678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:42.584220 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" event={"ID":"04df1b12-1787-4992-9563-104c12b5e2ac","Type":"ContainerDied","Data":"cd2d7d4e151a8d057bb3f4cd6132016a1c164a853cc91f25bfb6d4c92d7db588"} Oct 31 19:40:43.728249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.728223 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:43.734318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.734300 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89\" (UniqueName: \"kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89\") pod \"04df1b12-1787-4992-9563-104c12b5e2ac\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " Oct 31 19:40:43.734424 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.734349 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca\") pod \"04df1b12-1787-4992-9563-104c12b5e2ac\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " Oct 31 19:40:43.734424 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.734379 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca\") pod \"04df1b12-1787-4992-9563-104c12b5e2ac\" (UID: \"04df1b12-1787-4992-9563-104c12b5e2ac\") " Oct 31 19:40:43.734775 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.734750 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca" pod "04df1b12-1787-4992-9563-104c12b5e2ac" (UID: "04df1b12-1787-4992-9563-104c12b5e2ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:43.736436 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.736414 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89" pod "04df1b12-1787-4992-9563-104c12b5e2ac" (UID: "04df1b12-1787-4992-9563-104c12b5e2ac"). InnerVolumeSpecName "kube-api-access-fwt89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:43.736531 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.736461 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca" pod "04df1b12-1787-4992-9563-104c12b5e2ac" (UID: "04df1b12-1787-4992-9563-104c12b5e2ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:43.836100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.836078 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwt89\" (UniqueName: \"kubernetes.io/projected/04df1b12-1787-4992-9563-104c12b5e2ac-kube-api-access-fwt89\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:43.836100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.836100 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:43.836100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:43.836110 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df1b12-1787-4992-9563-104c12b5e2ac-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:40:44.595871 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:44.595841 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" Oct 31 19:40:44.596047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:44.595840 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx" event={"ID":"04df1b12-1787-4992-9563-104c12b5e2ac","Type":"ContainerDied","Data":"31a92d5644cd16544c33a375f4509e6476f7b3a3a9f1c5aaba948a34876015f0"} Oct 31 19:40:44.596047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:40:44.595959 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a92d5644cd16544c33a375f4509e6476f7b3a3a9f1c5aaba948a34876015f0" Oct 31 19:40:54.903716 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:40:54.903675 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:41:11.481069 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:11.480970 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf"] Oct 31 19:41:11.632423 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:11.632391 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq"] Oct 31 19:41:16.482281 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:16.482218 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" containerName="payload-processing" containerID="cri-o://827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8" gracePeriod=25 Oct 31 19:41:16.633114 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:16.633057 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerName="payload-pre-processing" containerID="cri-o://a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a" gracePeriod=25 Oct 31 19:41:19.506545 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:19.506501 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:41:19.506951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:19.506519 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:41:20.268319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:20.268294 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:41:20.273319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:20.273293 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:41:24.904970 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:41:24.904931 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:41:29.506092 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:29.506057 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:41:29.506513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:29.506109 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:41:36.765742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.765717 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:41:36.784649 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.784617 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7\" (UniqueName: \"kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7\") pod \"efd74f69-ecf0-419c-a821-a93c42903a38\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " Oct 31 19:41:36.784821 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.784688 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume\") pod \"efd74f69-ecf0-419c-a821-a93c42903a38\" (UID: \"efd74f69-ecf0-419c-a821-a93c42903a38\") " Oct 31 19:41:36.785164 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.785135 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume" pod "efd74f69-ecf0-419c-a821-a93c42903a38" (UID: "efd74f69-ecf0-419c-a821-a93c42903a38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:36.786980 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.786951 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7" pod "efd74f69-ecf0-419c-a821-a93c42903a38" (UID: "efd74f69-ecf0-419c-a821-a93c42903a38"). InnerVolumeSpecName "kube-api-access-mm5k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:36.880530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.880506 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:41:36.886570 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.886542 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm5k7\" (UniqueName: \"kubernetes.io/projected/efd74f69-ecf0-419c-a821-a93c42903a38-kube-api-access-mm5k7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:36.886570 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.886570 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd74f69-ecf0-419c-a821-a93c42903a38-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:36.909329 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.909298 5688 generic.go:356] "Generic (PLEG): container finished" podID="efd74f69-ecf0-419c-a821-a93c42903a38" containerID="827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8" exitCode=0 Oct 31 19:41:36.909471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.909346 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" Oct 31 19:41:36.909471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.909381 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" event={"ID":"efd74f69-ecf0-419c-a821-a93c42903a38","Type":"ContainerDied","Data":"827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8"} Oct 31 19:41:36.909471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.909431 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf" event={"ID":"efd74f69-ecf0-419c-a821-a93c42903a38","Type":"ContainerDied","Data":"8c4eb92023467370e94a6f9f1db978703318f151ef664cd8e0bfb3e484968b32"} Oct 31 19:41:36.909471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.909456 5688 scope.go:122] "RemoveContainer" containerID="827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8" Oct 31 19:41:36.911075 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.911050 5688 generic.go:356] "Generic (PLEG): container finished" podID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerID="a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a" exitCode=0 Oct 31 19:41:36.911266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.911090 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" Oct 31 19:41:36.911266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.911117 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" event={"ID":"d13087a8-45a8-4780-8f7a-480a745c9bd9","Type":"ContainerDied","Data":"a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a"} Oct 31 19:41:36.911266 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.911155 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq" event={"ID":"d13087a8-45a8-4780-8f7a-480a745c9bd9","Type":"ContainerDied","Data":"b264952863bcd957ca15bbb28aa3eb2d541c846665c87cb0e8fae0bbbdeb7e5b"} Oct 31 19:41:36.921320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.921301 5688 scope.go:122] "RemoveContainer" containerID="827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8" Oct 31 19:41:36.921603 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:41:36.921586 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8\": container with ID starting with 827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8 not found: ID does not exist" containerID="827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8" Oct 31 19:41:36.921657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.921610 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8"} err="failed to get container status \"827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8\": rpc error: code = NotFound desc = could not find container \"827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8\": container with ID starting with 827411f352895cb1b7823bac02e73b831b8a78258d851dca7a1a524c118c67d8 not found: ID does not exist" Oct 31 19:41:36.921657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.921625 5688 scope.go:122] "RemoveContainer" containerID="a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a" Oct 31 19:41:36.930039 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.930019 5688 scope.go:122] "RemoveContainer" containerID="a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a" Oct 31 19:41:36.930283 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:41:36.930268 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a\": container with ID starting with a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a not found: ID does not exist" containerID="a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a" Oct 31 19:41:36.930346 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.930290 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a"} err="failed to get container status \"a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a\": rpc error: code = NotFound desc = could not find container \"a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a\": container with ID starting with a92184129cd13ffce39678d2685fdfac3b590c0c711fe5e7ca22cc9c0303c59a not found: ID does not exist" Oct 31 19:41:36.987659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.987637 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9\" (UniqueName: \"kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9\") pod \"d13087a8-45a8-4780-8f7a-480a745c9bd9\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " Oct 31 19:41:36.987810 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.987721 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume\") pod \"d13087a8-45a8-4780-8f7a-480a745c9bd9\" (UID: \"d13087a8-45a8-4780-8f7a-480a745c9bd9\") " Oct 31 19:41:36.988091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.988067 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume" pod "d13087a8-45a8-4780-8f7a-480a745c9bd9" (UID: "d13087a8-45a8-4780-8f7a-480a745c9bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:36.989812 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:36.989788 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9" pod "d13087a8-45a8-4780-8f7a-480a745c9bd9" (UID: "d13087a8-45a8-4780-8f7a-480a745c9bd9"). InnerVolumeSpecName "kube-api-access-rjsf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:37.088652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.088579 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13087a8-45a8-4780-8f7a-480a745c9bd9-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:37.088652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.088605 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjsf9\" (UniqueName: \"kubernetes.io/projected/d13087a8-45a8-4780-8f7a-480a745c9bd9-kube-api-access-rjsf9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:37.163403 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.163371 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf"] Oct 31 19:41:37.166675 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.166645 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1aefcd-b9c9fcf55-87vbf"] Oct 31 19:41:37.343328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.343245 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq"] Oct 31 19:41:37.349951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.349912 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1aefcd-65c75c6495-dxwcq"] Oct 31 19:41:37.566279 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.566241 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" path="/var/lib/kubelet/pods/d13087a8-45a8-4780-8f7a-480a745c9bd9/volumes" Oct 31 19:41:37.566736 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:37.566720 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" path="/var/lib/kubelet/pods/efd74f69-ecf0-419c-a821-a93c42903a38/volumes" Oct 31 19:41:42.980958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.980920 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc"] Oct 31 19:41:42.981642 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981622 5688 cpu_manager.go:414] "containerMap: removing container" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" containerName="payload-processing" Oct 31 19:41:42.981738 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981645 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" containerName="payload-processing" Oct 31 19:41:42.981738 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981661 5688 cpu_manager.go:414] "containerMap: removing container" podUID="04df1b12-1787-4992-9563-104c12b5e2ac" containerName="revoke-keys" Oct 31 19:41:42.981738 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981668 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04df1b12-1787-4992-9563-104c12b5e2ac" containerName="revoke-keys" Oct 31 19:41:42.981738 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981702 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerName="payload-pre-processing" Oct 31 19:41:42.981738 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981711 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerName="payload-pre-processing" Oct 31 19:41:42.981989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981824 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13087a8-45a8-4780-8f7a-480a745c9bd9" containerName="payload-pre-processing" Oct 31 19:41:42.981989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981846 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd74f69-ecf0-419c-a821-a93c42903a38" containerName="payload-processing" Oct 31 19:41:42.981989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.981856 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="04df1b12-1787-4992-9563-104c12b5e2ac" containerName="revoke-keys" Oct 31 19:41:42.986404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:42.986383 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.041502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.041463 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:41:43.050932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.050909 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brmj\" (UniqueName: \"kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.051078 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.050945 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.051154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.051067 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.091229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.091202 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:41:43.093607 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.093585 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc"] Oct 31 19:41:43.151791 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.151733 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.151968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.151943 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brmj\" (UniqueName: \"kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.152115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.152099 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.152628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.152499 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.154560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.154541 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.209276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.209253 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brmj\" (UniqueName: \"kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj\") pod \"maas-api-revoke-sub-a37e974e-8vbwc\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.299367 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.299344 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:43.671965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.671942 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc"] Oct 31 19:41:43.673829 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:41:43.673802 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd61c01e_6e8c_4d6a_8d6c_c6f9d385cdde.slice/crio-4e5e46ac9acf8f62cd2be4d59db6ac8e6cd2e22d483257d376249a74d4996094 WatchSource:0}: Error finding container 4e5e46ac9acf8f62cd2be4d59db6ac8e6cd2e22d483257d376249a74d4996094: Status 404 returned error can't find the container with id 4e5e46ac9acf8f62cd2be4d59db6ac8e6cd2e22d483257d376249a74d4996094 Oct 31 19:41:43.953710 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.953680 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" event={"ID":"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde","Type":"ContainerStarted","Data":"de94e643d0579fdb7a580180161055d91f5ff942428bcdf5919b937087b9f17b"} Oct 31 19:41:43.953828 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:43.953740 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" event={"ID":"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde","Type":"ContainerStarted","Data":"4e5e46ac9acf8f62cd2be4d59db6ac8e6cd2e22d483257d376249a74d4996094"} Oct 31 19:41:44.079584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:44.079539 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" podStartSLOduration=2.079526176 podStartE2EDuration="2.079526176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:44.07684512 +0000 UTC m=+4824.678986265" watchObservedRunningTime="2026-10-31 19:41:44.079526176 +0000 UTC m=+4824.681667320" Oct 31 19:41:44.960950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:44.960871 5688 generic.go:356] "Generic (PLEG): container finished" podID="bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" containerID="de94e643d0579fdb7a580180161055d91f5ff942428bcdf5919b937087b9f17b" exitCode=0 Oct 31 19:41:44.960950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:44.960926 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" event={"ID":"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde","Type":"ContainerDied","Data":"de94e643d0579fdb7a580180161055d91f5ff942428bcdf5919b937087b9f17b"} Oct 31 19:41:46.110926 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.110903 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:46.185509 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.185486 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca\") pod \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " Oct 31 19:41:46.185634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.185544 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj\" (UniqueName: \"kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj\") pod \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " Oct 31 19:41:46.185634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.185586 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca\") pod \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\" (UID: \"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde\") " Oct 31 19:41:46.185978 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.185951 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca" pod "bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" (UID: "bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:46.187668 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.187649 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj" pod "bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" (UID: "bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde"). InnerVolumeSpecName "kube-api-access-4brmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:46.187668 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.187658 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca" pod "bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" (UID: "bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:41:46.287632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.287587 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:46.287632 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.287632 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brmj\" (UniqueName: \"kubernetes.io/projected/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-kube-api-access-4brmj\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:46.287922 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.287651 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:46.972440 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.972410 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" Oct 31 19:41:46.972609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.972442 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc" event={"ID":"bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde","Type":"ContainerDied","Data":"4e5e46ac9acf8f62cd2be4d59db6ac8e6cd2e22d483257d376249a74d4996094"} Oct 31 19:41:46.972609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:46.972496 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5e46ac9acf8f62cd2be4d59db6ac8e6cd2e22d483257d376249a74d4996094" Oct 31 19:41:50.319825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.319789 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm"] Oct 31 19:41:50.320324 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.319988 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" podUID="6ff64189-a47e-41b0-9438-fad7681635b6" containerName="maas-api" containerID="cri-o://0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606" gracePeriod=30 Oct 31 19:41:50.618869 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.618845 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:41:50.732904 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.732873 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca\") pod \"6ff64189-a47e-41b0-9438-fad7681635b6\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " Oct 31 19:41:50.733170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.732919 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m\" (UniqueName: \"kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m\") pod \"6ff64189-a47e-41b0-9438-fad7681635b6\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " Oct 31 19:41:50.733170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.733059 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs\") pod \"6ff64189-a47e-41b0-9438-fad7681635b6\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " Oct 31 19:41:50.733170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.733104 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\") pod \"6ff64189-a47e-41b0-9438-fad7681635b6\" (UID: \"6ff64189-a47e-41b0-9438-fad7681635b6\") " Oct 31 19:41:50.733416 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.733242 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca" pod "6ff64189-a47e-41b0-9438-fad7681635b6" (UID: "6ff64189-a47e-41b0-9438-fad7681635b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:50.733513 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.733488 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff64189-a47e-41b0-9438-fad7681635b6-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:50.735835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.735798 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs" pod "6ff64189-a47e-41b0-9438-fad7681635b6" (UID: "6ff64189-a47e-41b0-9438-fad7681635b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:41:50.736081 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.736059 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls" pod "6ff64189-a47e-41b0-9438-fad7681635b6" (UID: "6ff64189-a47e-41b0-9438-fad7681635b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:41:50.736398 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.736372 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m" pod "6ff64189-a47e-41b0-9438-fad7681635b6" (UID: "6ff64189-a47e-41b0-9438-fad7681635b6"). InnerVolumeSpecName "kube-api-access-nkh6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.834087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.834051 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkh6m\" (UniqueName: \"kubernetes.io/projected/6ff64189-a47e-41b0-9438-fad7681635b6-kube-api-access-nkh6m\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:50.834087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.834080 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:50.834273 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.834094 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff64189-a47e-41b0-9438-fad7681635b6-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:50.995800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.995766 5688 generic.go:356] "Generic (PLEG): container finished" podID="6ff64189-a47e-41b0-9438-fad7681635b6" containerID="0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606" exitCode=0 Oct 31 19:41:50.995967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.995813 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" Oct 31 19:41:50.995967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.995829 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" event={"ID":"6ff64189-a47e-41b0-9438-fad7681635b6","Type":"ContainerDied","Data":"0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606"} Oct 31 19:41:50.995967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.995869 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm" event={"ID":"6ff64189-a47e-41b0-9438-fad7681635b6","Type":"ContainerDied","Data":"86b2152c3b798ef8193b047ec7479119fa70bdf82669e8dec5f70e79c56e1ec9"} Oct 31 19:41:50.995967 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:50.995890 5688 scope.go:122] "RemoveContainer" containerID="0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606" Oct 31 19:41:51.008627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:51.008606 5688 scope.go:122] "RemoveContainer" containerID="0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606" Oct 31 19:41:51.008973 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:41:51.008954 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606\": container with ID starting with 0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606 not found: ID does not exist" containerID="0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606" Oct 31 19:41:51.009036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:51.008983 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606"} err="failed to get container status \"0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606\": rpc error: code = NotFound desc = could not find container \"0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606\": container with ID starting with 0a8b85d4555e65fbcb6b74f5f877a75c70f7f9f14db5d591704f2ec97bbe9606 not found: ID does not exist" Oct 31 19:41:51.159950 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:51.159918 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm"] Oct 31 19:41:51.165204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:51.165156 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1aefcd-78b9bcddc4-dxgsm"] Oct 31 19:41:51.565714 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:51.565678 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff64189-a47e-41b0-9438-fad7681635b6" path="/var/lib/kubelet/pods/6ff64189-a47e-41b0-9438-fad7681635b6/volumes" Oct 31 19:41:54.797297 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.797259 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp"] Oct 31 19:41:54.797854 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.797833 5688 cpu_manager.go:414] "containerMap: removing container" podUID="bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" containerName="revoke-keys" Oct 31 19:41:54.797960 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.797857 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" containerName="revoke-keys" Oct 31 19:41:54.797960 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.797906 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6ff64189-a47e-41b0-9438-fad7681635b6" containerName="maas-api" Oct 31 19:41:54.797960 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.797916 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff64189-a47e-41b0-9438-fad7681635b6" containerName="maas-api" Oct 31 19:41:54.798115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.798046 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff64189-a47e-41b0-9438-fad7681635b6" containerName="maas-api" Oct 31 19:41:54.798115 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.798065 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" containerName="revoke-keys" Oct 31 19:41:54.802704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.802679 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.857299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.857275 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp"] Oct 31 19:41:54.873222 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.873178 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.873338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.873257 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.873338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.873316 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rz7\" (UniqueName: \"kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.904467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.904431 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:41:54.904631 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.904520 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:41:54.905257 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:41:54.905233 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:41:54.974066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.974038 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rz7\" (UniqueName: \"kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.974216 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.974172 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.974276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.974236 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.974861 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.974838 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:54.976550 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:54.976533 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:55.030745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:55.030720 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rz7\" (UniqueName: \"kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7\") pod \"maas-api-revoke-sub-a3170b0a-hjtgp\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:55.115862 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:55.115792 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:55.487122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:55.487094 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp"] Oct 31 19:41:55.488720 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:41:55.488696 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad994526_68cf_4e96_a8bc_963c2cc9a328.slice/crio-5c4c7a8fa65b3bd321b7c006cfb61055bab5efb45484b1ca74d2d51d80aaec53 WatchSource:0}: Error finding container 5c4c7a8fa65b3bd321b7c006cfb61055bab5efb45484b1ca74d2d51d80aaec53: Status 404 returned error can't find the container with id 5c4c7a8fa65b3bd321b7c006cfb61055bab5efb45484b1ca74d2d51d80aaec53 Oct 31 19:41:56.025390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:56.025294 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" event={"ID":"ad994526-68cf-4e96-a8bc-963c2cc9a328","Type":"ContainerStarted","Data":"00369669d9385fb320f3d5f4b9ab5e5f7c231cec3c3d24f3b755cdb056f4a889"} Oct 31 19:41:56.025390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:56.025354 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" event={"ID":"ad994526-68cf-4e96-a8bc-963c2cc9a328","Type":"ContainerStarted","Data":"5c4c7a8fa65b3bd321b7c006cfb61055bab5efb45484b1ca74d2d51d80aaec53"} Oct 31 19:41:56.142343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:56.142302 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" podStartSLOduration=2.142289046 podStartE2EDuration="2.142289046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:56.140471971 +0000 UTC m=+4836.742613114" watchObservedRunningTime="2026-10-31 19:41:56.142289046 +0000 UTC m=+4836.744430190" Oct 31 19:41:57.040659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:57.040624 5688 generic.go:356] "Generic (PLEG): container finished" podID="ad994526-68cf-4e96-a8bc-963c2cc9a328" containerID="00369669d9385fb320f3d5f4b9ab5e5f7c231cec3c3d24f3b755cdb056f4a889" exitCode=0 Oct 31 19:41:57.041109 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:57.040709 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" event={"ID":"ad994526-68cf-4e96-a8bc-963c2cc9a328","Type":"ContainerDied","Data":"00369669d9385fb320f3d5f4b9ab5e5f7c231cec3c3d24f3b755cdb056f4a889"} Oct 31 19:41:58.190609 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.190577 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:58.316804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.316722 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca\") pod \"ad994526-68cf-4e96-a8bc-963c2cc9a328\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " Oct 31 19:41:58.316965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.316829 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca\") pod \"ad994526-68cf-4e96-a8bc-963c2cc9a328\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " Oct 31 19:41:58.316965 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.316865 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7\" (UniqueName: \"kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7\") pod \"ad994526-68cf-4e96-a8bc-963c2cc9a328\" (UID: \"ad994526-68cf-4e96-a8bc-963c2cc9a328\") " Oct 31 19:41:58.317565 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.317538 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca" pod "ad994526-68cf-4e96-a8bc-963c2cc9a328" (UID: "ad994526-68cf-4e96-a8bc-963c2cc9a328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:58.319074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.319048 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca" pod "ad994526-68cf-4e96-a8bc-963c2cc9a328" (UID: "ad994526-68cf-4e96-a8bc-963c2cc9a328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:41:58.319753 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.319717 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7" pod "ad994526-68cf-4e96-a8bc-963c2cc9a328" (UID: "ad994526-68cf-4e96-a8bc-963c2cc9a328"). InnerVolumeSpecName "kube-api-access-d4rz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:58.418422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.418391 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:58.418422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.418418 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rz7\" (UniqueName: \"kubernetes.io/projected/ad994526-68cf-4e96-a8bc-963c2cc9a328-kube-api-access-d4rz7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:58.418422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:58.418428 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad994526-68cf-4e96-a8bc-963c2cc9a328-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:41:59.056380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:59.056343 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" event={"ID":"ad994526-68cf-4e96-a8bc-963c2cc9a328","Type":"ContainerDied","Data":"5c4c7a8fa65b3bd321b7c006cfb61055bab5efb45484b1ca74d2d51d80aaec53"} Oct 31 19:41:59.056380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:59.056381 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp" Oct 31 19:41:59.056597 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:41:59.056400 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4c7a8fa65b3bd321b7c006cfb61055bab5efb45484b1ca74d2d51d80aaec53" Oct 31 19:42:02.193144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.193116 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws"] Oct 31 19:42:02.193591 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.193576 5688 cpu_manager.go:414] "containerMap: removing container" podUID="ad994526-68cf-4e96-a8bc-963c2cc9a328" containerName="revoke-keys" Oct 31 19:42:02.193651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.193591 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad994526-68cf-4e96-a8bc-963c2cc9a328" containerName="revoke-keys" Oct 31 19:42:02.193716 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.193707 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad994526-68cf-4e96-a8bc-963c2cc9a328" containerName="revoke-keys" Oct 31 19:42:02.197052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.197038 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.253988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.253961 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws"] Oct 31 19:42:02.300280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.300251 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:42:02.300280 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.300258 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:42:02.362081 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.362053 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.362221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.362118 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.362276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.362232 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzc86\" (UniqueName: \"kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.462847 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.462784 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzc86\" (UniqueName: \"kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.462972 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.462855 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.462972 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.462899 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.463497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.463476 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.465361 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.465343 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.519558 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.519527 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzc86\" (UniqueName: \"kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86\") pod \"maas-api-revoke-sub-78fb452d-tcpws\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:02.809702 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:02.809679 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:03.185971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:03.185936 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws"] Oct 31 19:42:03.187563 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:42:03.187533 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10faf1a3_f32e_4f54_b1c3_db6aa8f7edcd.slice/crio-f767caa210786a8b22e19d58a6cad0bba8a9670eabc933a662be0d391ea84057 WatchSource:0}: Error finding container f767caa210786a8b22e19d58a6cad0bba8a9670eabc933a662be0d391ea84057: Status 404 returned error can't find the container with id f767caa210786a8b22e19d58a6cad0bba8a9670eabc933a662be0d391ea84057 Oct 31 19:42:04.090567 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:04.090537 5688 generic.go:356] "Generic (PLEG): container finished" podID="10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" containerID="827c46dde4f5c7582425e4abeca2a1a493a95576b947d55b5446f78d53198d6e" exitCode=0 Oct 31 19:42:04.090968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:04.090618 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" event={"ID":"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd","Type":"ContainerDied","Data":"827c46dde4f5c7582425e4abeca2a1a493a95576b947d55b5446f78d53198d6e"} Oct 31 19:42:04.090968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:04.090717 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" event={"ID":"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd","Type":"ContainerStarted","Data":"f767caa210786a8b22e19d58a6cad0bba8a9670eabc933a662be0d391ea84057"} Oct 31 19:42:05.243439 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.243416 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:05.402818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.402742 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86\" (UniqueName: \"kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86\") pod \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " Oct 31 19:42:05.402973 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.402951 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca\") pod \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " Oct 31 19:42:05.403028 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.402991 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca\") pod \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\" (UID: \"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd\") " Oct 31 19:42:05.403388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.403367 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca" pod "10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" (UID: "10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:05.405071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.405047 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca" pod "10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" (UID: "10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:05.405243 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.405211 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86" pod "10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" (UID: "10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd"). InnerVolumeSpecName "kube-api-access-hzc86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:05.503812 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.503785 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:05.503812 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.503812 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:05.503959 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:05.503823 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzc86\" (UniqueName: \"kubernetes.io/projected/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd-kube-api-access-hzc86\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:06.102821 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.102792 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" Oct 31 19:42:06.103002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.102791 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws" event={"ID":"10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd","Type":"ContainerDied","Data":"f767caa210786a8b22e19d58a6cad0bba8a9670eabc933a662be0d391ea84057"} Oct 31 19:42:06.103002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.102915 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f767caa210786a8b22e19d58a6cad0bba8a9670eabc933a662be0d391ea84057" Oct 31 19:42:06.422058 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.421990 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r"] Oct 31 19:42:06.422463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.422439 5688 cpu_manager.go:414] "containerMap: removing container" podUID="10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" containerName="revoke-keys" Oct 31 19:42:06.422463 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.422452 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" containerName="revoke-keys" Oct 31 19:42:06.422568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.422558 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" containerName="revoke-keys" Oct 31 19:42:06.425663 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.425644 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.477977 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.477954 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:42:06.478111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.477954 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:42:06.498122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.498094 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r"] Oct 31 19:42:06.574562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.574533 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j"] Oct 31 19:42:06.584469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.584441 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94"] Oct 31 19:42:06.594091 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.594067 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h"] Oct 31 19:42:06.614466 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.614441 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2skl\" (UniqueName: \"kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.614586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.614485 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.614586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.614549 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.715989 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.715403 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2skl\" (UniqueName: \"kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.716231 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.715992 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.716231 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.716052 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.723747 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.723710 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.724944 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.724917 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:06.772860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:06.772829 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2skl\" (UniqueName: \"kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl\") pod \"maas-api-revoke-sub-47884d9a-jlt7r\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:07.038423 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:07.038391 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:07.410339 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:07.410314 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r"] Oct 31 19:42:07.412558 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:42:07.412527 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebb54ac_af21_446f_9847_c5da68cf7e2f.slice/crio-b57a1f6a858451f4d51360487f79b2073d438f0c17d08b8528ced9774ab1a75f WatchSource:0}: Error finding container b57a1f6a858451f4d51360487f79b2073d438f0c17d08b8528ced9774ab1a75f: Status 404 returned error can't find the container with id b57a1f6a858451f4d51360487f79b2073d438f0c17d08b8528ced9774ab1a75f Oct 31 19:42:08.117034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:08.116959 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" event={"ID":"eebb54ac-af21-446f-9847-c5da68cf7e2f","Type":"ContainerStarted","Data":"b9828241eb0b9745109a5118531a9fbba996596c1cb3747d52e237f365470f6c"} Oct 31 19:42:08.117034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:08.117018 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" event={"ID":"eebb54ac-af21-446f-9847-c5da68cf7e2f","Type":"ContainerStarted","Data":"b57a1f6a858451f4d51360487f79b2073d438f0c17d08b8528ced9774ab1a75f"} Oct 31 19:42:08.233022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:08.232973 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" podStartSLOduration=2.232959075 podStartE2EDuration="2.232959075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:08.231557983 +0000 UTC m=+4848.833699128" watchObservedRunningTime="2026-10-31 19:42:08.232959075 +0000 UTC m=+4848.835100257" Oct 31 19:42:09.125055 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:09.125016 5688 generic.go:356] "Generic (PLEG): container finished" podID="eebb54ac-af21-446f-9847-c5da68cf7e2f" containerID="b9828241eb0b9745109a5118531a9fbba996596c1cb3747d52e237f365470f6c" exitCode=0 Oct 31 19:42:09.125444 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:09.125108 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" event={"ID":"eebb54ac-af21-446f-9847-c5da68cf7e2f","Type":"ContainerDied","Data":"b9828241eb0b9745109a5118531a9fbba996596c1cb3747d52e237f365470f6c"} Oct 31 19:42:10.267066 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.267044 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:10.350956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.350930 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca\") pod \"eebb54ac-af21-446f-9847-c5da68cf7e2f\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " Oct 31 19:42:10.351063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.351030 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca\") pod \"eebb54ac-af21-446f-9847-c5da68cf7e2f\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " Oct 31 19:42:10.351111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.351066 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl\" (UniqueName: \"kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl\") pod \"eebb54ac-af21-446f-9847-c5da68cf7e2f\" (UID: \"eebb54ac-af21-446f-9847-c5da68cf7e2f\") " Oct 31 19:42:10.351370 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.351347 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca" pod "eebb54ac-af21-446f-9847-c5da68cf7e2f" (UID: "eebb54ac-af21-446f-9847-c5da68cf7e2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:10.353072 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.353040 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca" pod "eebb54ac-af21-446f-9847-c5da68cf7e2f" (UID: "eebb54ac-af21-446f-9847-c5da68cf7e2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:10.353168 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.353097 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl" pod "eebb54ac-af21-446f-9847-c5da68cf7e2f" (UID: "eebb54ac-af21-446f-9847-c5da68cf7e2f"). InnerVolumeSpecName "kube-api-access-r2skl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:10.451839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.451774 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:10.451839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.451799 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebb54ac-af21-446f-9847-c5da68cf7e2f-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:10.451839 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:10.451809 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2skl\" (UniqueName: \"kubernetes.io/projected/eebb54ac-af21-446f-9847-c5da68cf7e2f-kube-api-access-r2skl\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:11.139041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:11.139009 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" event={"ID":"eebb54ac-af21-446f-9847-c5da68cf7e2f","Type":"ContainerDied","Data":"b57a1f6a858451f4d51360487f79b2073d438f0c17d08b8528ced9774ab1a75f"} Oct 31 19:42:11.139041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:11.139036 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r" Oct 31 19:42:11.139299 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:11.139061 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b57a1f6a858451f4d51360487f79b2073d438f0c17d08b8528ced9774ab1a75f" Oct 31 19:42:17.389395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.389357 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c"] Oct 31 19:42:17.390011 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.389993 5688 cpu_manager.go:414] "containerMap: removing container" podUID="eebb54ac-af21-446f-9847-c5da68cf7e2f" containerName="revoke-keys" Oct 31 19:42:17.390097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.390015 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eebb54ac-af21-446f-9847-c5da68cf7e2f" containerName="revoke-keys" Oct 31 19:42:17.390201 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.390165 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="eebb54ac-af21-446f-9847-c5da68cf7e2f" containerName="revoke-keys" Oct 31 19:42:17.393627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.393606 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.409733 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.409707 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwdm\" (UniqueName: \"kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.409835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.409753 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.409901 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.409883 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.497703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.497678 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:42:17.497703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.497696 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:42:17.500497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.500473 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c"] Oct 31 19:42:17.510955 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.510931 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.511053 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.511005 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwdm\" (UniqueName: \"kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.511053 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.511046 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.511561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.511542 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.513538 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.513516 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.568274 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.568252 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwdm\" (UniqueName: \"kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm\") pod \"maas-api-revoke-sub-d44186b3-smd9c\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:17.706745 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:17.706694 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:18.085235 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:18.082738 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c"] Oct 31 19:42:18.192124 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:18.192092 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" event={"ID":"f290f8d4-6993-4b36-a8f7-f16f9285fa9f","Type":"ContainerStarted","Data":"d52e157bc1957a536b51b2ff58e3a98c476cb3f8f6583b72d2b6ffe8efe15d25"} Oct 31 19:42:18.990908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:18.990879 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq"] Oct 31 19:42:18.994545 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:18.994530 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.025221 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.025194 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.025323 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.025263 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq69p\" (UniqueName: \"kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.025422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.025403 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.051696 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.051675 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq"] Oct 31 19:42:19.126769 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.126693 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq69p\" (UniqueName: \"kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.126920 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.126785 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.126920 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.126842 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.127435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.127413 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.129253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.129233 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.183236 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.183206 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq69p\" (UniqueName: \"kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p\") pod \"maas-api-revoke-sub-0c9071f1-rrvkq\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.198852 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.198828 5688 generic.go:356] "Generic (PLEG): container finished" podID="f290f8d4-6993-4b36-a8f7-f16f9285fa9f" containerID="fccff50f3062efbe0e3c392ca233cb92c598c5ae0666a8c8962e2332e15f30ee" exitCode=0 Oct 31 19:42:19.198958 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.198903 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" event={"ID":"f290f8d4-6993-4b36-a8f7-f16f9285fa9f","Type":"ContainerDied","Data":"fccff50f3062efbe0e3c392ca233cb92c598c5ae0666a8c8962e2332e15f30ee"} Oct 31 19:42:19.307603 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.307573 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:19.689420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:19.689388 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq"] Oct 31 19:42:19.705656 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:42:19.705622 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a601b4_a673_47c4_97cf_680866392d09.slice/crio-82c7e3a2a5db26d86e7a355abd38a48cb294ebae4e40d12716cf1ea2b18a6b00 WatchSource:0}: Error finding container 82c7e3a2a5db26d86e7a355abd38a48cb294ebae4e40d12716cf1ea2b18a6b00: Status 404 returned error can't find the container with id 82c7e3a2a5db26d86e7a355abd38a48cb294ebae4e40d12716cf1ea2b18a6b00 Oct 31 19:42:20.205883 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.205851 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" event={"ID":"99a601b4-a673-47c4-97cf-680866392d09","Type":"ContainerStarted","Data":"8302cfa689af8354666e9f9722938a79ae414556ec5f87430c9a64368dfd2c08"} Oct 31 19:42:20.206269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.205912 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" event={"ID":"99a601b4-a673-47c4-97cf-680866392d09","Type":"ContainerStarted","Data":"82c7e3a2a5db26d86e7a355abd38a48cb294ebae4e40d12716cf1ea2b18a6b00"} Oct 31 19:42:20.322132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.321961 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" podStartSLOduration=2.321947372 podStartE2EDuration="2.321947372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:20.31966777 +0000 UTC m=+4860.921808915" watchObservedRunningTime="2026-10-31 19:42:20.321947372 +0000 UTC m=+4860.924088516" Oct 31 19:42:20.357142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.357116 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:20.439389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.439362 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca\") pod \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " Oct 31 19:42:20.439533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.439427 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm\" (UniqueName: \"kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm\") pod \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " Oct 31 19:42:20.439533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.439500 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca\") pod \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\" (UID: \"f290f8d4-6993-4b36-a8f7-f16f9285fa9f\") " Oct 31 19:42:20.439761 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.439736 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca" pod "f290f8d4-6993-4b36-a8f7-f16f9285fa9f" (UID: "f290f8d4-6993-4b36-a8f7-f16f9285fa9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:20.439924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.439901 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:20.441511 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.441491 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca" pod "f290f8d4-6993-4b36-a8f7-f16f9285fa9f" (UID: "f290f8d4-6993-4b36-a8f7-f16f9285fa9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:20.441589 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.441547 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm" pod "f290f8d4-6993-4b36-a8f7-f16f9285fa9f" (UID: "f290f8d4-6993-4b36-a8f7-f16f9285fa9f"). InnerVolumeSpecName "kube-api-access-xdwdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:20.541468 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.541440 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:20.541468 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:20.541464 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwdm\" (UniqueName: \"kubernetes.io/projected/f290f8d4-6993-4b36-a8f7-f16f9285fa9f-kube-api-access-xdwdm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.212530 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.212441 5688 generic.go:356] "Generic (PLEG): container finished" podID="99a601b4-a673-47c4-97cf-680866392d09" containerID="8302cfa689af8354666e9f9722938a79ae414556ec5f87430c9a64368dfd2c08" exitCode=0 Oct 31 19:42:21.212934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.212526 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" event={"ID":"99a601b4-a673-47c4-97cf-680866392d09","Type":"ContainerDied","Data":"8302cfa689af8354666e9f9722938a79ae414556ec5f87430c9a64368dfd2c08"} Oct 31 19:42:21.213939 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.213925 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" Oct 31 19:42:21.214030 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.213934 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c" event={"ID":"f290f8d4-6993-4b36-a8f7-f16f9285fa9f","Type":"ContainerDied","Data":"d52e157bc1957a536b51b2ff58e3a98c476cb3f8f6583b72d2b6ffe8efe15d25"} Oct 31 19:42:21.214030 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.213974 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52e157bc1957a536b51b2ff58e3a98c476cb3f8f6583b72d2b6ffe8efe15d25" Oct 31 19:42:21.595910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.595860 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" podUID="21ca53f6-8a76-475a-8d2e-f87789d8cb9c" containerName="main" containerID="cri-o://f61971f3521e53ed9f12cf4d544a68f035c753c39e4fec42a6b0cea1927673a0" gracePeriod=45 Oct 31 19:42:21.608581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.608545 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" containerID="cri-o://6e36144e0b7dc5ba7344d1e696e1f62c8e048842d7c4de803fb3547c6ea38a04" gracePeriod=45 Oct 31 19:42:21.616547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.616519 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" containerID="cri-o://dc80df2beeb35082da40d5142811b0454796eaa52f78c1621771da3e8053667a" gracePeriod=45 Oct 31 19:42:21.798678 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.798636 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": dial tcp 10.132.0.44:8000: connect: connection refused" Oct 31 19:42:21.877311 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.877286 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 19:42:21.913709 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.913688 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 19:42:21.918038 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.918019 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 19:42:21.958259 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958220 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home\") pod \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " Oct 31 19:42:21.958404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958315 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache\") pod \"641de09a-8b10-4de2-a672-e0a751b1c646\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " Oct 31 19:42:21.958404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958346 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir\") pod \"641de09a-8b10-4de2-a672-e0a751b1c646\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " Oct 31 19:42:21.958404 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958393 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs\") pod \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " Oct 31 19:42:21.958584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958420 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache\") pod \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " Oct 31 19:42:21.958584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958426 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home" pod "21ca53f6-8a76-475a-8d2e-f87789d8cb9c" (UID: "21ca53f6-8a76-475a-8d2e-f87789d8cb9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.958584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958458 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm\") pod \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958579 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm\") pod \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958559 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache" pod "641de09a-8b10-4de2-a672-e0a751b1c646" (UID: "641de09a-8b10-4de2-a672-e0a751b1c646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958610 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9\" (UniqueName: \"kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9\") pod \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958631 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir" pod "641de09a-8b10-4de2-a672-e0a751b1c646" (UID: "641de09a-8b10-4de2-a672-e0a751b1c646"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958651 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home\") pod \"641de09a-8b10-4de2-a672-e0a751b1c646\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958703 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache" pod "21ca53f6-8a76-475a-8d2e-f87789d8cb9c" (UID: "21ca53f6-8a76-475a-8d2e-f87789d8cb9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.958749 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958736 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache\") pod \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958780 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm\") pod \"641de09a-8b10-4de2-a672-e0a751b1c646\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958833 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664\" (UniqueName: \"kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664\") pod \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958855 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home" pod "641de09a-8b10-4de2-a672-e0a751b1c646" (UID: "641de09a-8b10-4de2-a672-e0a751b1c646"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958869 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir\") pod \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.958985 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache" pod "0dde1a57-750a-4b21-9530-8e9917cf6b3a" (UID: "0dde1a57-750a-4b21-9530-8e9917cf6b3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959053 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir" pod "0dde1a57-750a-4b21-9530-8e9917cf6b3a" (UID: "0dde1a57-750a-4b21-9530-8e9917cf6b3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959199 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home\") pod \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\" (UID: \"0dde1a57-750a-4b21-9530-8e9917cf6b3a\") " Oct 31 19:42:21.959363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959237 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5\" (UniqueName: \"kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5\") pod \"641de09a-8b10-4de2-a672-e0a751b1c646\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " Oct 31 19:42:21.959796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959438 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home" pod "0dde1a57-750a-4b21-9530-8e9917cf6b3a" (UID: "0dde1a57-750a-4b21-9530-8e9917cf6b3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.959796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959466 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs\" (UniqueName: \"kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs\") pod \"641de09a-8b10-4de2-a672-e0a751b1c646\" (UID: \"641de09a-8b10-4de2-a672-e0a751b1c646\") " Oct 31 19:42:21.959796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959515 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs\") pod \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " Oct 31 19:42:21.959796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959551 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir\") pod \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\" (UID: \"21ca53f6-8a76-475a-8d2e-f87789d8cb9c\") " Oct 31 19:42:21.960002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959959 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959981 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960002 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.959994 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.960007 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.960021 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.960033 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.960046 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960161 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.960059 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:21.960439 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.960274 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir" pod "21ca53f6-8a76-475a-8d2e-f87789d8cb9c" (UID: "21ca53f6-8a76-475a-8d2e-f87789d8cb9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.961401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.961374 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm" pod "0dde1a57-750a-4b21-9530-8e9917cf6b3a" (UID: "0dde1a57-750a-4b21-9530-8e9917cf6b3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.961507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.961476 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs" pod "0dde1a57-750a-4b21-9530-8e9917cf6b3a" (UID: "0dde1a57-750a-4b21-9530-8e9917cf6b3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:21.961507 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.961489 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm" pod "21ca53f6-8a76-475a-8d2e-f87789d8cb9c" (UID: "21ca53f6-8a76-475a-8d2e-f87789d8cb9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.961636 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.961587 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9" pod "21ca53f6-8a76-475a-8d2e-f87789d8cb9c" (UID: "21ca53f6-8a76-475a-8d2e-f87789d8cb9c"). InnerVolumeSpecName "kube-api-access-phcw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:21.961696 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.961681 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664" pod "0dde1a57-750a-4b21-9530-8e9917cf6b3a" (UID: "0dde1a57-750a-4b21-9530-8e9917cf6b3a"). InnerVolumeSpecName "kube-api-access-jn664". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:21.962358 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.962334 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5" pod "641de09a-8b10-4de2-a672-e0a751b1c646" (UID: "641de09a-8b10-4de2-a672-e0a751b1c646"). InnerVolumeSpecName "kube-api-access-ksjt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:21.962533 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.962511 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs" pod "641de09a-8b10-4de2-a672-e0a751b1c646" (UID: "641de09a-8b10-4de2-a672-e0a751b1c646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:21.962660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.962643 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm" pod "641de09a-8b10-4de2-a672-e0a751b1c646" (UID: "641de09a-8b10-4de2-a672-e0a751b1c646"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:21.963079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:21.963064 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs" pod "21ca53f6-8a76-475a-8d2e-f87789d8cb9c" (UID: "21ca53f6-8a76-475a-8d2e-f87789d8cb9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:22.009916 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.009889 5688 scope.go:122] "RemoveContainer" containerID="6e36144e0b7dc5ba7344d1e696e1f62c8e048842d7c4de803fb3547c6ea38a04" Oct 31 19:42:22.020583 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.020564 5688 scope.go:122] "RemoveContainer" containerID="dc80df2beeb35082da40d5142811b0454796eaa52f78c1621771da3e8053667a" Oct 31 19:42:22.031439 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.031421 5688 scope.go:122] "RemoveContainer" containerID="f61971f3521e53ed9f12cf4d544a68f035c753c39e4fec42a6b0cea1927673a0" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061103 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dde1a57-750a-4b21-9530-8e9917cf6b3a-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061124 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061132 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dde1a57-750a-4b21-9530-8e9917cf6b3a-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061140 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phcw9\" (UniqueName: \"kubernetes.io/projected/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-kube-api-access-phcw9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061152 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641de09a-8b10-4de2-a672-e0a751b1c646-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061164 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn664\" (UniqueName: \"kubernetes.io/projected/0dde1a57-750a-4b21-9530-8e9917cf6b3a-kube-api-access-jn664\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061173 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksjt5\" (UniqueName: \"kubernetes.io/projected/641de09a-8b10-4de2-a672-e0a751b1c646-kube-api-access-ksjt5\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061254 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641de09a-8b10-4de2-a672-e0a751b1c646-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061265 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.062903 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.061272 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ca53f6-8a76-475a-8d2e-f87789d8cb9c-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.220401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220319 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" event={"ID":"0dde1a57-750a-4b21-9530-8e9917cf6b3a","Type":"ContainerDied","Data":"dc80df2beeb35082da40d5142811b0454796eaa52f78c1621771da3e8053667a"} Oct 31 19:42:22.220401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220371 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" event={"ID":"0dde1a57-750a-4b21-9530-8e9917cf6b3a","Type":"ContainerDied","Data":"70bb715292823a5f985f8ac4d193aebf848179418f476b8e51240a4fa257f8e1"} Oct 31 19:42:22.220401 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220388 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h" Oct 31 19:42:22.220933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220391 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" event={"ID":"21ca53f6-8a76-475a-8d2e-f87789d8cb9c","Type":"ContainerDied","Data":"f61971f3521e53ed9f12cf4d544a68f035c753c39e4fec42a6b0cea1927673a0"} Oct 31 19:42:22.220933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220543 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" event={"ID":"21ca53f6-8a76-475a-8d2e-f87789d8cb9c","Type":"ContainerDied","Data":"ac1c641cc5ffcca6a4199774cb117e9578155263790d71bbe1a0fcce8ba3a416"} Oct 31 19:42:22.220933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220566 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" Oct 31 19:42:22.220933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220575 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" event={"ID":"641de09a-8b10-4de2-a672-e0a751b1c646","Type":"ContainerDied","Data":"6e36144e0b7dc5ba7344d1e696e1f62c8e048842d7c4de803fb3547c6ea38a04"} Oct 31 19:42:22.220933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220593 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94" event={"ID":"641de09a-8b10-4de2-a672-e0a751b1c646","Type":"ContainerDied","Data":"71ed91a792011181cc14a9abfd7ba05316bb4a58cb6635afc6bb083315f4f4fd"} Oct 31 19:42:22.220933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.220629 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" Oct 31 19:42:22.350385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.350361 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:22.365020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.364998 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p\" (UniqueName: \"kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p\") pod \"99a601b4-a673-47c4-97cf-680866392d09\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " Oct 31 19:42:22.365156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.365139 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca\") pod \"99a601b4-a673-47c4-97cf-680866392d09\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " Oct 31 19:42:22.365238 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.365209 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca\") pod \"99a601b4-a673-47c4-97cf-680866392d09\" (UID: \"99a601b4-a673-47c4-97cf-680866392d09\") " Oct 31 19:42:22.365494 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.365462 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca" pod "99a601b4-a673-47c4-97cf-680866392d09" (UID: "99a601b4-a673-47c4-97cf-680866392d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:22.367319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.367295 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca" pod "99a601b4-a673-47c4-97cf-680866392d09" (UID: "99a601b4-a673-47c4-97cf-680866392d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:22.367414 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.367320 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p" pod "99a601b4-a673-47c4-97cf-680866392d09" (UID: "99a601b4-a673-47c4-97cf-680866392d09"). InnerVolumeSpecName "kube-api-access-fq69p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:22.447956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.447926 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j"] Oct 31 19:42:22.454439 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.454417 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j"] Oct 31 19:42:22.466436 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.466417 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq69p\" (UniqueName: \"kubernetes.io/projected/99a601b4-a673-47c4-97cf-680866392d09-kube-api-access-fq69p\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.466436 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.466436 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a601b4-a673-47c4-97cf-680866392d09-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.466582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.466448 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a601b4-a673-47c4-97cf-680866392d09-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:42:22.631048 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.631021 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h"] Oct 31 19:42:22.634734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.634713 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/simulator-w4-6279f5-kserve-796c58b6b4-pp85h"] Oct 31 19:42:22.807433 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.807407 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94"] Oct 31 19:42:22.809393 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.809373 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/premium-w4-6279f5-kserve-76d79fc69c-t2d94"] Oct 31 19:42:22.833047 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:22.833001 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6279f5/distinct-w4-6279f5-kserve-5944c6d77b-rfn6j" podUID="21ca53f6-8a76-475a-8d2e-f87789d8cb9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 19:42:23.227483 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:23.227455 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" Oct 31 19:42:23.227862 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:23.227492 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq" event={"ID":"99a601b4-a673-47c4-97cf-680866392d09","Type":"ContainerDied","Data":"82c7e3a2a5db26d86e7a355abd38a48cb294ebae4e40d12716cf1ea2b18a6b00"} Oct 31 19:42:23.227862 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:23.227542 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c7e3a2a5db26d86e7a355abd38a48cb294ebae4e40d12716cf1ea2b18a6b00" Oct 31 19:42:23.565910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:23.565879 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" path="/var/lib/kubelet/pods/0dde1a57-750a-4b21-9530-8e9917cf6b3a/volumes" Oct 31 19:42:23.566294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:23.566280 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ca53f6-8a76-475a-8d2e-f87789d8cb9c" path="/var/lib/kubelet/pods/21ca53f6-8a76-475a-8d2e-f87789d8cb9c/volumes" Oct 31 19:42:23.566630 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:23.566619 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" path="/var/lib/kubelet/pods/641de09a-8b10-4de2-a672-e0a751b1c646/volumes" Oct 31 19:42:24.904838 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:42:24.904805 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:42:54.904309 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:42:54.904274 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:42:59.334639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:59.334606 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc"] Oct 31 19:42:59.335011 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:42:59.334836 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" containerID="cri-o://9836adb9ac24cfa7d8377ace8bdc81a3a73c0048c3020456c249faf24d687830" gracePeriod=30 Oct 31 19:43:04.468543 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.468514 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf"] Oct 31 19:43:04.494140 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.494107 5688 generic.go:356] "Generic (PLEG): container finished" podID="70e00ec3-6702-4512-9022-c7e604564478" containerID="9836adb9ac24cfa7d8377ace8bdc81a3a73c0048c3020456c249faf24d687830" exitCode=0 Oct 31 19:43:04.494336 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.494307 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" event={"ID":"70e00ec3-6702-4512-9022-c7e604564478","Type":"ContainerDied","Data":"9836adb9ac24cfa7d8377ace8bdc81a3a73c0048c3020456c249faf24d687830"} Oct 31 19:43:04.494405 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.494376 5688 scope.go:122] "RemoveContainer" containerID="b21e278dcec06a5c2cd6fa0a724841b452153533181873a5981595331d563d04" Oct 31 19:43:04.611840 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.611784 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 19:43:04.696986 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.696960 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697176 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697021 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697176 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697090 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697176 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697140 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697213 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697267 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:43:04.697399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697325 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697303 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:43:04.697399 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697373 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697424 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697438 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:43:04.697646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697451 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy\") pod \"70e00ec3-6702-4512-9022-c7e604564478\" (UID: \"70e00ec3-6702-4512-9022-c7e604564478\") " Oct 31 19:43:04.697646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697533 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:43:04.697646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697636 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:43:04.697844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697816 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.697844 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697837 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.697954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697850 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e00ec3-6702-4512-9022-c7e604564478-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.697954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697863 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.697954 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.697879 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.699488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.699463 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:43:04.699616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.699566 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "kube-api-access-q9b8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:43:04.699711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.699690 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:43:04.699711 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.699695 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo" pod "70e00ec3-6702-4512-9022-c7e604564478" (UID: "70e00ec3-6702-4512-9022-c7e604564478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:43:04.799295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.799257 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.799295 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.799293 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9b8s\" (UniqueName: \"kubernetes.io/projected/70e00ec3-6702-4512-9022-c7e604564478-kube-api-access-q9b8s\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.799489 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.799308 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e00ec3-6702-4512-9022-c7e604564478-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:04.799489 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:04.799321 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e00ec3-6702-4512-9022-c7e604564478-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:05.501671 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:05.501643 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" Oct 31 19:43:05.502100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:05.501636 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc" event={"ID":"70e00ec3-6702-4512-9022-c7e604564478","Type":"ContainerDied","Data":"1c02753696ecac543f3984cd2968b2f1d550556e9d98b8c54b0d324883ef05d4"} Oct 31 19:43:05.502100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:05.501785 5688 scope.go:122] "RemoveContainer" containerID="9836adb9ac24cfa7d8377ace8bdc81a3a73c0048c3020456c249faf24d687830" Oct 31 19:43:05.671832 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:05.671803 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc"] Oct 31 19:43:05.675458 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:05.675433 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-zn8vc"] Oct 31 19:43:07.565052 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:07.565016 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e00ec3-6702-4512-9022-c7e604564478" path="/var/lib/kubelet/pods/70e00ec3-6702-4512-9022-c7e604564478/volumes" Oct 31 19:43:09.469591 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:09.469554 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerName="payload-processing" containerID="cri-o://d3a13c83fc7d0c8b62f252f4241867730b1af71c642975114a32d75c96d37b33" gracePeriod=25 Oct 31 19:43:18.101422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:18.101385 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:43:24.903534 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:43:24.903501 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:43:25.277205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:25.277152 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck"] Oct 31 19:43:27.049034 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:27.049000 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5"] Oct 31 19:43:27.098313 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:27.098282 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd966ca1-8wgx5"] Oct 31 19:43:27.566075 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:27.566040 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c09fd91-63a7-4699-a611-61e6e7038a5e" path="/var/lib/kubelet/pods/6c09fd91-63a7-4699-a611-61e6e7038a5e/volumes" Oct 31 19:43:28.102123 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:28.102079 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:43:29.631634 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.631604 5688 generic.go:356] "Generic (PLEG): container finished" podID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerID="d3a13c83fc7d0c8b62f252f4241867730b1af71c642975114a32d75c96d37b33" exitCode=0 Oct 31 19:43:29.632020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.631677 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" event={"ID":"5d0ede4c-e957-4863-8939-0d7ef964c2f9","Type":"ContainerDied","Data":"d3a13c83fc7d0c8b62f252f4241867730b1af71c642975114a32d75c96d37b33"} Oct 31 19:43:29.726945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.726888 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:43:29.866253 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.866227 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn\" (UniqueName: \"kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn\") pod \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " Oct 31 19:43:29.866409 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.866283 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume\") pod \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\" (UID: \"5d0ede4c-e957-4863-8939-0d7ef964c2f9\") " Oct 31 19:43:29.866735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.866708 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume" pod "5d0ede4c-e957-4863-8939-0d7ef964c2f9" (UID: "5d0ede4c-e957-4863-8939-0d7ef964c2f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:43:29.868386 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.868360 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn" pod "5d0ede4c-e957-4863-8939-0d7ef964c2f9" (UID: "5d0ede4c-e957-4863-8939-0d7ef964c2f9"). InnerVolumeSpecName "kube-api-access-5kdcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:43:29.967773 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.967745 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdcn\" (UniqueName: \"kubernetes.io/projected/5d0ede4c-e957-4863-8939-0d7ef964c2f9-kube-api-access-5kdcn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:29.967869 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:29.967775 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0ede4c-e957-4863-8939-0d7ef964c2f9-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:30.278023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:30.277988 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerName="payload-processing" containerID="cri-o://00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503" gracePeriod=25 Oct 31 19:43:30.638291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:30.638219 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" Oct 31 19:43:30.638291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:30.638230 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf" event={"ID":"5d0ede4c-e957-4863-8939-0d7ef964c2f9","Type":"ContainerDied","Data":"4da70cac734fdd265f414a4c162a634bb59799d38aedbf76273906f1a99217a3"} Oct 31 19:43:30.638743 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:30.638296 5688 scope.go:122] "RemoveContainer" containerID="d3a13c83fc7d0c8b62f252f4241867730b1af71c642975114a32d75c96d37b33" Oct 31 19:43:30.807646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:30.807610 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf"] Oct 31 19:43:30.811730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:30.811709 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbceb9-9f6994d59-6jxxf"] Oct 31 19:43:31.564607 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:31.564573 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" path="/var/lib/kubelet/pods/5d0ede4c-e957-4863-8939-0d7ef964c2f9/volumes" Oct 31 19:43:36.201134 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:36.201098 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:43:37.046438 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:37.046408 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb"] Oct 31 19:43:37.095932 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:37.095904 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798d20cc-685jb"] Oct 31 19:43:37.565022 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:37.564988 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8779577-f7a0-428c-9ec2-a0e572f04c9c" path="/var/lib/kubelet/pods/e8779577-f7a0-428c-9ec2-a0e572f04c9c/volumes" Oct 31 19:43:46.200627 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:46.200585 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:43:49.074539 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:49.074505 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7"] Oct 31 19:43:49.124154 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:49.124114 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd201420-ttcb7"] Oct 31 19:43:49.565023 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:49.564992 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482b3eaa-1b55-4d3d-b315-9c0b5a5234ca" path="/var/lib/kubelet/pods/482b3eaa-1b55-4d3d-b315-9c0b5a5234ca/volumes" Oct 31 19:43:50.538651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.538591 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:43:50.670018 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.669987 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume\") pod \"884dde44-39ce-4e1b-b7e1-c59d07f01556\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " Oct 31 19:43:50.670213 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.670079 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82\" (UniqueName: \"kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82\") pod \"884dde44-39ce-4e1b-b7e1-c59d07f01556\" (UID: \"884dde44-39ce-4e1b-b7e1-c59d07f01556\") " Oct 31 19:43:50.670508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.670479 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume" pod "884dde44-39ce-4e1b-b7e1-c59d07f01556" (UID: "884dde44-39ce-4e1b-b7e1-c59d07f01556"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:43:50.672587 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.672557 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82" pod "884dde44-39ce-4e1b-b7e1-c59d07f01556" (UID: "884dde44-39ce-4e1b-b7e1-c59d07f01556"). InnerVolumeSpecName "kube-api-access-b8c82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:43:50.759878 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.759849 5688 generic.go:356] "Generic (PLEG): container finished" podID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerID="00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503" exitCode=0 Oct 31 19:43:50.760037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.759885 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" event={"ID":"884dde44-39ce-4e1b-b7e1-c59d07f01556","Type":"ContainerDied","Data":"00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503"} Oct 31 19:43:50.760037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.759898 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" Oct 31 19:43:50.760037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.759920 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck" event={"ID":"884dde44-39ce-4e1b-b7e1-c59d07f01556","Type":"ContainerDied","Data":"f3c7d76da35152c935e60cb1f6942c1229c123012aeae029f4503b4448f32d46"} Oct 31 19:43:50.760037 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.759942 5688 scope.go:122] "RemoveContainer" containerID="00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503" Oct 31 19:43:50.769651 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.769624 5688 scope.go:122] "RemoveContainer" containerID="00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503" Oct 31 19:43:50.770060 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:43:50.770030 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503\": container with ID starting with 00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503 not found: ID does not exist" containerID="00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503" Oct 31 19:43:50.770171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.770069 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503"} err="failed to get container status \"00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503\": rpc error: code = NotFound desc = could not find container \"00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503\": container with ID starting with 00b007802a497bac7c23b02cbd45cba0e0fd00fbfab9e400e8df15ca1dc75503 not found: ID does not exist" Oct 31 19:43:50.771684 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.771657 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884dde44-39ce-4e1b-b7e1-c59d07f01556-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:50.771779 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.771690 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8c82\" (UniqueName: \"kubernetes.io/projected/884dde44-39ce-4e1b-b7e1-c59d07f01556-kube-api-access-b8c82\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:43:50.925227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.925124 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck"] Oct 31 19:43:50.929545 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:50.929501 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6cfd68-db8d6f989-72kck"] Oct 31 19:43:51.565746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:51.565713 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" path="/var/lib/kubelet/pods/884dde44-39ce-4e1b-b7e1-c59d07f01556/volumes" Oct 31 19:43:54.903440 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:43:54.903402 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:43:55.715995 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:55.715961 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl"] Oct 31 19:43:55.766946 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:55.766903 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x"] Oct 31 19:43:55.770200 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:43:55.770159 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr"] Oct 31 19:44:00.074759 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:00.074681 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt"] Oct 31 19:44:00.123021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:00.122990 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bede1cbc-7xmzt"] Oct 31 19:44:01.565074 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:01.565042 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec98734-9195-4438-ac02-cf6631535085" path="/var/lib/kubelet/pods/4ec98734-9195-4438-ac02-cf6631535085/volumes" Oct 31 19:44:10.736263 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:10.736206 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" containerID="cri-o://fd83f9137e779293350321abac7bdb2218691c642a228b76780be21082d00bbe" gracePeriod=45 Oct 31 19:44:10.787755 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:10.787709 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerName="main" containerID="cri-o://8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33" gracePeriod=45 Oct 31 19:44:10.793529 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:10.793497 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerName="main" containerID="cri-o://4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987" gracePeriod=45 Oct 31 19:44:10.834065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:10.834040 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": dial tcp 10.132.0.75:8000: connect: connection refused" Oct 31 19:44:10.869879 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:10.869859 5688 generic.go:356] "Generic (PLEG): container finished" podID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerID="fd83f9137e779293350321abac7bdb2218691c642a228b76780be21082d00bbe" exitCode=0 Oct 31 19:44:10.869969 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:10.869931 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" event={"ID":"03854b8e-b51c-4f0a-9126-23b1bf099a15","Type":"ContainerDied","Data":"fd83f9137e779293350321abac7bdb2218691c642a228b76780be21082d00bbe"} Oct 31 19:44:11.052438 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.052414 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 19:44:11.079664 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.079644 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 19:44:11.087318 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.087298 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 19:44:11.153058 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153029 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm\") pod \"03854b8e-b51c-4f0a-9126-23b1bf099a15\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " Oct 31 19:44:11.153307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153111 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x\" (UniqueName: \"kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x\") pod \"03854b8e-b51c-4f0a-9126-23b1bf099a15\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " Oct 31 19:44:11.153307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153152 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache\") pod \"03854b8e-b51c-4f0a-9126-23b1bf099a15\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " Oct 31 19:44:11.153307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153199 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home\") pod \"03854b8e-b51c-4f0a-9126-23b1bf099a15\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " Oct 31 19:44:11.153307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153242 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs\" (UniqueName: \"kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs\") pod \"03854b8e-b51c-4f0a-9126-23b1bf099a15\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " Oct 31 19:44:11.153560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153310 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs\") pod \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " Oct 31 19:44:11.153560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153348 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv\" (UniqueName: \"kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv\") pod \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " Oct 31 19:44:11.153560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153377 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir\") pod \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " Oct 31 19:44:11.153560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153436 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache\") pod \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " Oct 31 19:44:11.153560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153500 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm\") pod \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " Oct 31 19:44:11.153560 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153542 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home\") pod \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\" (UID: \"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad\") " Oct 31 19:44:11.153876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.153610 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir\") pod \"03854b8e-b51c-4f0a-9126-23b1bf099a15\" (UID: \"03854b8e-b51c-4f0a-9126-23b1bf099a15\") " Oct 31 19:44:11.154357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.154022 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache" pod "03854b8e-b51c-4f0a-9126-23b1bf099a15" (UID: "03854b8e-b51c-4f0a-9126-23b1bf099a15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.154357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.154036 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache" pod "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" (UID: "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.154357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.154138 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir" pod "03854b8e-b51c-4f0a-9126-23b1bf099a15" (UID: "03854b8e-b51c-4f0a-9126-23b1bf099a15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.154357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.154315 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir" pod "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" (UID: "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.154357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.154332 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home" pod "03854b8e-b51c-4f0a-9126-23b1bf099a15" (UID: "03854b8e-b51c-4f0a-9126-23b1bf099a15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.154357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.154340 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home" pod "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" (UID: "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.156935 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.156908 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs" pod "03854b8e-b51c-4f0a-9126-23b1bf099a15" (UID: "03854b8e-b51c-4f0a-9126-23b1bf099a15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:44:11.157059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.156911 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs" pod "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" (UID: "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:44:11.157132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.157054 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x" pod "03854b8e-b51c-4f0a-9126-23b1bf099a15" (UID: "03854b8e-b51c-4f0a-9126-23b1bf099a15"). InnerVolumeSpecName "kube-api-access-b5t8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:11.157132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.157076 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv" pod "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" (UID: "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad"). InnerVolumeSpecName "kube-api-access-vnktv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:11.157321 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.157167 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm" pod "03854b8e-b51c-4f0a-9126-23b1bf099a15" (UID: "03854b8e-b51c-4f0a-9126-23b1bf099a15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.157389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.157356 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm" pod "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" (UID: "1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.254510 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254435 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs\") pod \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " Oct 31 19:44:11.254510 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254506 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home\") pod \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " Oct 31 19:44:11.254694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254553 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir\") pod \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " Oct 31 19:44:11.254694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254611 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp\" (UniqueName: \"kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp\") pod \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " Oct 31 19:44:11.254694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254648 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache\") pod \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " Oct 31 19:44:11.254694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254670 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm\") pod \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\" (UID: \"2b6e10e4-0667-4d3f-996f-32b87a52b5de\") " Oct 31 19:44:11.254914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254767 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home" pod "2b6e10e4-0667-4d3f-996f-32b87a52b5de" (UID: "2b6e10e4-0667-4d3f-996f-32b87a52b5de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.254914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254786 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir" pod "2b6e10e4-0667-4d3f-996f-32b87a52b5de" (UID: "2b6e10e4-0667-4d3f-996f-32b87a52b5de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.255027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.254910 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache" pod "2b6e10e4-0667-4d3f-996f-32b87a52b5de" (UID: "2b6e10e4-0667-4d3f-996f-32b87a52b5de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.255027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255014 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnktv\" (UniqueName: \"kubernetes.io/projected/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-kube-api-access-vnktv\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255034 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255049 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255063 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255075 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255088 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255100 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255114 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255128 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255142 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255140 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5t8x\" (UniqueName: \"kubernetes.io/projected/03854b8e-b51c-4f0a-9126-23b1bf099a15-kube-api-access-b5t8x\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255153 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255165 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03854b8e-b51c-4f0a-9126-23b1bf099a15-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255202 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03854b8e-b51c-4f0a-9126-23b1bf099a15-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255215 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.255517 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.255227 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.256605 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.256582 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs" pod "2b6e10e4-0667-4d3f-996f-32b87a52b5de" (UID: "2b6e10e4-0667-4d3f-996f-32b87a52b5de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:44:11.256662 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.256637 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm" pod "2b6e10e4-0667-4d3f-996f-32b87a52b5de" (UID: "2b6e10e4-0667-4d3f-996f-32b87a52b5de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:11.256746 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.256732 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp" pod "2b6e10e4-0667-4d3f-996f-32b87a52b5de" (UID: "2b6e10e4-0667-4d3f-996f-32b87a52b5de"). InnerVolumeSpecName "kube-api-access-mgwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:11.355941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.355910 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6e10e4-0667-4d3f-996f-32b87a52b5de-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.355941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.355940 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwbp\" (UniqueName: \"kubernetes.io/projected/2b6e10e4-0667-4d3f-996f-32b87a52b5de-kube-api-access-mgwbp\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.356143 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.355953 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6e10e4-0667-4d3f-996f-32b87a52b5de-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:11.876549 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.876484 5688 generic.go:356] "Generic (PLEG): container finished" podID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerID="8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33" exitCode=0 Oct 31 19:44:11.876549 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.876524 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" Oct 31 19:44:11.876966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.876566 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" event={"ID":"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad","Type":"ContainerDied","Data":"8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33"} Oct 31 19:44:11.876966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.876604 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x" event={"ID":"1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad","Type":"ContainerDied","Data":"ba1ccbe059d62a8845dfa212bf5779be719bccbd7390ffc27222eccf2a7474d9"} Oct 31 19:44:11.876966 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.876624 5688 scope.go:122] "RemoveContainer" containerID="8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33" Oct 31 19:44:11.878363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.878338 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" event={"ID":"03854b8e-b51c-4f0a-9126-23b1bf099a15","Type":"ContainerDied","Data":"384912f63a30a4dd41af98f1f1c9b8e7619c3f5721699222e5bfbf66d57ecead"} Oct 31 19:44:11.878480 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.878379 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl" Oct 31 19:44:11.880050 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.880025 5688 generic.go:356] "Generic (PLEG): container finished" podID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerID="4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987" exitCode=0 Oct 31 19:44:11.880156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.880093 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" event={"ID":"2b6e10e4-0667-4d3f-996f-32b87a52b5de","Type":"ContainerDied","Data":"4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987"} Oct 31 19:44:11.880156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.880121 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" event={"ID":"2b6e10e4-0667-4d3f-996f-32b87a52b5de","Type":"ContainerDied","Data":"5f610f2dc9f861962916115a678af2e98b3d7879d057a9f50e962a0f8fd4b374"} Oct 31 19:44:11.880305 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.880219 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr" Oct 31 19:44:11.887001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.886985 5688 scope.go:122] "RemoveContainer" containerID="8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33" Oct 31 19:44:11.887307 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:11.887286 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33\": container with ID starting with 8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33 not found: ID does not exist" containerID="8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33" Oct 31 19:44:11.887420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.887313 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33"} err="failed to get container status \"8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33\": rpc error: code = NotFound desc = could not find container \"8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33\": container with ID starting with 8ecc45ff51b2dd810986326aee2a1e559151e1872807d03ecae43cd09d9e9c33 not found: ID does not exist" Oct 31 19:44:11.887420 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.887331 5688 scope.go:122] "RemoveContainer" containerID="fd83f9137e779293350321abac7bdb2218691c642a228b76780be21082d00bbe" Oct 31 19:44:11.895156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.895132 5688 scope.go:122] "RemoveContainer" containerID="4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987" Oct 31 19:44:11.902586 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.902571 5688 scope.go:122] "RemoveContainer" containerID="4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987" Oct 31 19:44:11.902864 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:11.902840 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987\": container with ID starting with 4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987 not found: ID does not exist" containerID="4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987" Oct 31 19:44:11.902988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:11.902871 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987"} err="failed to get container status \"4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987\": rpc error: code = NotFound desc = could not find container \"4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987\": container with ID starting with 4c3f5314c0c281d250f1adb9d845817f425d881c73c3538b991d54d63f447987 not found: ID does not exist" Oct 31 19:44:12.051345 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:12.051323 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl"] Oct 31 19:44:12.099860 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:12.099835 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-2-w2-88085e-kserve-5ccdf96fd-nfhsl"] Oct 31 19:44:12.186952 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:12.186898 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x"] Oct 31 19:44:12.191436 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:12.191414 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/premium-w2-88085e-kserve-d68487646-gbs8x"] Oct 31 19:44:12.302291 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:12.302257 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr"] Oct 31 19:44:12.307255 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:12.307235 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/simulator-w2-88085e-kserve-797bd57d75-stnkr"] Oct 31 19:44:13.571628 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:13.571594 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" path="/var/lib/kubelet/pods/03854b8e-b51c-4f0a-9126-23b1bf099a15/volumes" Oct 31 19:44:13.572019 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:13.571967 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" path="/var/lib/kubelet/pods/1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad/volumes" Oct 31 19:44:13.572343 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:13.572329 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" path="/var/lib/kubelet/pods/2b6e10e4-0667-4d3f-996f-32b87a52b5de/volumes" Oct 31 19:44:17.072949 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:17.072917 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc"] Oct 31 19:44:17.122395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:17.122365 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3b325f-5xbmc"] Oct 31 19:44:17.565168 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:17.565135 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fcb598-2d84-4b86-aa01-e32958043126" path="/var/lib/kubelet/pods/79fcb598-2d84-4b86-aa01-e32958043126/volumes" Oct 31 19:44:24.903334 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:24.903299 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:44:26.074482 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:26.074450 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg"] Oct 31 19:44:26.124110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:26.124083 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b9ca8b-zgdtg"] Oct 31 19:44:27.565139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:27.565106 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166aa54d-69fa-4bdb-b85d-72bfc894d235" path="/var/lib/kubelet/pods/166aa54d-69fa-4bdb-b85d-72bfc894d235/volumes" Oct 31 19:44:34.046488 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.046457 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr"] Oct 31 19:44:34.096670 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.096638 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254a4e70-5gzxr"] Oct 31 19:44:34.602152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.602122 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85"] Oct 31 19:44:34.602377 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.602337 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" containerID="cri-o://bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117" gracePeriod=30 Oct 31 19:44:34.653664 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.653633 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr"] Oct 31 19:44:34.654089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.654059 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" containerID="cri-o://303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52" gracePeriod=30 Oct 31 19:44:34.656774 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.656737 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5"] Oct 31 19:44:34.656974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:34.656955 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" containerID="cri-o://6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10" gracePeriod=30 Oct 31 19:44:35.274561 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:35.274525 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-m9q85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:44:35.274960 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:35.274594 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:44:35.388428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:35.388393 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6zdwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:44:35.388608 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:35.388466 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:44:35.564968 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:35.564902 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af41b22-6920-4509-8d1d-9eaba237d680" path="/var/lib/kubelet/pods/4af41b22-6920-4509-8d1d-9eaba237d680/volumes" Oct 31 19:44:39.875898 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.875842 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 19:44:39.904735 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.904708 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.904877 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.904757 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.904877 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.904847 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.904991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.904879 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.904991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.904927 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.904991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.904979 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.905156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905012 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.905156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905020 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:39.905156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905052 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.905156 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905085 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket\") pod \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\" (UID: \"f1093ffb-21b2-4373-89eb-a99ed8ea88b3\") " Oct 31 19:44:39.905413 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905398 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:39.905935 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905913 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:39.906041 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.905970 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:39.906129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.906111 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:44:39.906225 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.906178 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:39.907941 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.907919 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:39.908249 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.908225 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "kube-api-access-c4l4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:39.908842 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.908813 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:39.909742 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.909716 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo" pod "f1093ffb-21b2-4373-89eb-a99ed8ea88b3" (UID: "f1093ffb-21b2-4373-89eb-a99ed8ea88b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:44:39.943395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.943374 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:44:39.972666 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:39.972647 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:44:40.005995 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.005967 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006014 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006062 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006110 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006096 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006157 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006217 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006259 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006309 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006293 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006321 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006352 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006379 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006412 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006415 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006442 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006492 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006567 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006603 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006635 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data\") pod \"cba88761-0d6b-4c65-a4a9-856b9d83a171\" (UID: \"cba88761-0d6b-4c65-a4a9-856b9d83a171\") " Oct 31 19:44:40.006921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006665 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs\") pod \"63ea3028-c752-4997-8cff-5c0f8ad16fca\" (UID: \"63ea3028-c752-4997-8cff-5c0f8ad16fca\") " Oct 31 19:44:40.006921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.006729 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:44:40.007305 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.007135 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:44:40.007389 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.007332 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.007472 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.007394 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.007896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.007599 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.007896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.007836 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.008079 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008058 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.008308 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008285 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.008456 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008434 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008570 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008502 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008652 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008636 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008655 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba88761-0d6b-4c65-a4a9-856b9d83a171-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008670 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008684 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008698 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ea3028-c752-4997-8cff-5c0f8ad16fca-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008712 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008725 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008739 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-kube-api-access-c4l4p\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008755 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008771 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008785 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008800 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1093ffb-21b2-4373-89eb-a99ed8ea88b3-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008813 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008826 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.008873 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.008841 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.010120 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010094 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "kube-api-access-wppzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:40.010265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010139 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:44:40.010265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010145 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:40.010265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010162 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "kube-api-access-s56r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:40.010265 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010238 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.010547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010359 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.010547 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010477 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:40.010834 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010814 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo" pod "63ea3028-c752-4997-8cff-5c0f8ad16fca" (UID: "63ea3028-c752-4997-8cff-5c0f8ad16fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:44:40.010997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.010983 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy" pod "cba88761-0d6b-4c65-a4a9-856b9d83a171" (UID: "cba88761-0d6b-4c65-a4a9-856b9d83a171"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:44:40.057532 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.057500 5688 generic.go:356] "Generic (PLEG): container finished" podID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerID="303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52" exitCode=0 Oct 31 19:44:40.057699 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.057614 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" Oct 31 19:44:40.057699 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.057635 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" event={"ID":"cba88761-0d6b-4c65-a4a9-856b9d83a171","Type":"ContainerDied","Data":"303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52"} Oct 31 19:44:40.057699 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.057684 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr" event={"ID":"cba88761-0d6b-4c65-a4a9-856b9d83a171","Type":"ContainerDied","Data":"ef51a527f5a8549a1a8c0407fefb60b42da792cf3ef43e0c68fb08e2378aef83"} Oct 31 19:44:40.057880 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.057711 5688 scope.go:122] "RemoveContainer" containerID="303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52" Oct 31 19:44:40.059582 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.059557 5688 generic.go:356] "Generic (PLEG): container finished" podID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerID="6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10" exitCode=0 Oct 31 19:44:40.059703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.059608 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" Oct 31 19:44:40.059703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.059651 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" event={"ID":"f1093ffb-21b2-4373-89eb-a99ed8ea88b3","Type":"ContainerDied","Data":"6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10"} Oct 31 19:44:40.059836 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.059700 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5" event={"ID":"f1093ffb-21b2-4373-89eb-a99ed8ea88b3","Type":"ContainerDied","Data":"f5781ee8b88c3c03a5c3c70b764f76b62a48450c9d7cf7f8c575e702290352eb"} Oct 31 19:44:40.063827 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.063797 5688 generic.go:356] "Generic (PLEG): container finished" podID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerID="bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117" exitCode=0 Oct 31 19:44:40.063942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.063841 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" Oct 31 19:44:40.063942 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.063912 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" event={"ID":"63ea3028-c752-4997-8cff-5c0f8ad16fca","Type":"ContainerDied","Data":"bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117"} Oct 31 19:44:40.064065 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.063945 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85" event={"ID":"63ea3028-c752-4997-8cff-5c0f8ad16fca","Type":"ContainerDied","Data":"59e8d1fdca0cff9b0e90e1db9072baf0028d9fc9102d1b4506a2164c8e684c40"} Oct 31 19:44:40.068139 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.068059 5688 scope.go:122] "RemoveContainer" containerID="303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52" Oct 31 19:44:40.068375 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:40.068356 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52\": container with ID starting with 303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52 not found: ID does not exist" containerID="303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52" Oct 31 19:44:40.068461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.068387 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52"} err="failed to get container status \"303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52\": rpc error: code = NotFound desc = could not find container \"303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52\": container with ID starting with 303f63e8e08da0fd4b54d548d6f8875afc180d085092e02044490236fe97cb52 not found: ID does not exist" Oct 31 19:44:40.068461 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.068409 5688 scope.go:122] "RemoveContainer" containerID="6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10" Oct 31 19:44:40.079499 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.079484 5688 scope.go:122] "RemoveContainer" containerID="6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10" Oct 31 19:44:40.079798 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:40.079770 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10\": container with ID starting with 6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10 not found: ID does not exist" containerID="6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10" Oct 31 19:44:40.079857 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.079807 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10"} err="failed to get container status \"6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10\": rpc error: code = NotFound desc = could not find container \"6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10\": container with ID starting with 6c82a7a0d06f1636a34c4e9552a355def30efc32e636dfd536e5732d756ebd10 not found: ID does not exist" Oct 31 19:44:40.079857 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.079825 5688 scope.go:122] "RemoveContainer" containerID="bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117" Oct 31 19:44:40.088385 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.088364 5688 scope.go:122] "RemoveContainer" containerID="bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117" Oct 31 19:44:40.088672 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:40.088654 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117\": container with ID starting with bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117 not found: ID does not exist" containerID="bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117" Oct 31 19:44:40.088733 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.088681 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117"} err="failed to get container status \"bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117\": rpc error: code = NotFound desc = could not find container \"bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117\": container with ID starting with bf371b2f5fb7a4816a04ab2dfff28d6e3332ed350910754e9f52b081f91bc117 not found: ID does not exist" Oct 31 19:44:40.109725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109707 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppzp\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-kube-api-access-wppzp\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109727 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109736 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109745 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109754 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109762 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109771 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba88761-0d6b-4c65-a4a9-856b9d83a171-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109779 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ea3028-c752-4997-8cff-5c0f8ad16fca-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.109846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.109787 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56r2\" (UniqueName: \"kubernetes.io/projected/cba88761-0d6b-4c65-a4a9-856b9d83a171-kube-api-access-s56r2\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:44:40.466125 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.466092 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5"] Oct 31 19:44:40.473642 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.473615 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lqzg5"] Oct 31 19:44:40.583132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.583104 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85"] Oct 31 19:44:40.587669 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.587648 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-m9q85"] Oct 31 19:44:40.697763 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.697736 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr"] Oct 31 19:44:40.704757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:40.704734 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6zdwr"] Oct 31 19:44:41.565390 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:41.565360 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" path="/var/lib/kubelet/pods/63ea3028-c752-4997-8cff-5c0f8ad16fca/volumes" Oct 31 19:44:41.565781 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:41.565772 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" path="/var/lib/kubelet/pods/cba88761-0d6b-4c65-a4a9-856b9d83a171/volumes" Oct 31 19:44:41.566166 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:44:41.566152 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" path="/var/lib/kubelet/pods/f1093ffb-21b2-4373-89eb-a99ed8ea88b3/volumes" Oct 31 19:44:54.903721 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:44:54.903686 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:45:00.161342 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161306 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75"] Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161661 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161671 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161678 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerName="main" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161684 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerName="main" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161696 5688 cpu_manager.go:414] "containerMap: removing container" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161700 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161708 5688 cpu_manager.go:414] "containerMap: removing container" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161713 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161725 5688 cpu_manager.go:414] "containerMap: removing container" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161729 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161740 5688 cpu_manager.go:414] "containerMap: removing container" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerName="payload-processing" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161746 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerName="payload-processing" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161752 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerName="payload-processing" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161757 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerName="payload-processing" Oct 31 19:45:00.161757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161763 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f290f8d4-6993-4b36-a8f7-f16f9285fa9f" containerName="revoke-keys" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161767 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f290f8d4-6993-4b36-a8f7-f16f9285fa9f" containerName="revoke-keys" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161777 5688 cpu_manager.go:414] "containerMap: removing container" podUID="21ca53f6-8a76-475a-8d2e-f87789d8cb9c" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161781 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ca53f6-8a76-475a-8d2e-f87789d8cb9c" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161789 5688 cpu_manager.go:414] "containerMap: removing container" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161793 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161803 5688 cpu_manager.go:414] "containerMap: removing container" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161807 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161815 5688 cpu_manager.go:414] "containerMap: removing container" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161820 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161828 5688 cpu_manager.go:414] "containerMap: removing container" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161832 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161840 5688 cpu_manager.go:414] "containerMap: removing container" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161844 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161852 5688 cpu_manager.go:414] "containerMap: removing container" podUID="99a601b4-a673-47c4-97cf-680866392d09" containerName="revoke-keys" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161856 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a601b4-a673-47c4-97cf-680866392d09" containerName="revoke-keys" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161909 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ea3028-c752-4997-8cff-5c0f8ad16fca" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161917 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="884dde44-39ce-4e1b-b7e1-c59d07f01556" containerName="payload-processing" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161923 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0ede4c-e957-4863-8939-0d7ef964c2f9" containerName="payload-processing" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161928 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f290f8d4-6993-4b36-a8f7-f16f9285fa9f" containerName="revoke-keys" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161934 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ca53f6-8a76-475a-8d2e-f87789d8cb9c" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161939 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="641de09a-8b10-4de2-a672-e0a751b1c646" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161944 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1093ffb-21b2-4373-89eb-a99ed8ea88b3" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161949 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6e10e4-0667-4d3f-996f-32b87a52b5de" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161955 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161960 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a601b4-a673-47c4-97cf-680866392d09" containerName="revoke-keys" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161965 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dde1a57-750a-4b21-9530-8e9917cf6b3a" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161970 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dabdcc5-d3c9-4eb6-a396-941b73b9d3ad" containerName="main" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161974 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba88761-0d6b-4c65-a4a9-856b9d83a171" containerName="istio-proxy" Oct 31 19:45:00.162322 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.161979 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="03854b8e-b51c-4f0a-9126-23b1bf099a15" containerName="main" Oct 31 19:45:00.164919 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.164900 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.217987 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.217944 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\"" Oct 31 19:45:00.269794 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.269763 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75"] Oct 31 19:45:00.280969 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.280940 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvgd\" (UniqueName: \"kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.281100 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.281004 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.381639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.381612 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvgd\" (UniqueName: \"kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.381758 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.381663 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.382250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.382233 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.440677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.440618 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvgd\" (UniqueName: \"kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.477595 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.477572 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:00.853137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:00.853049 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75"] Oct 31 19:45:00.855486 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:45:00.855456 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cb081a_061d_4dde_b191_390175ff3d4f.slice/crio-5fcdd46f1b030fbc8f02294c67dcb67d60d160895ca7fb1ba10de9a4ace96bca WatchSource:0}: Error finding container 5fcdd46f1b030fbc8f02294c67dcb67d60d160895ca7fb1ba10de9a4ace96bca: Status 404 returned error can't find the container with id 5fcdd46f1b030fbc8f02294c67dcb67d60d160895ca7fb1ba10de9a4ace96bca Oct 31 19:45:01.173348 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:01.173321 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerStarted","Data":"5f137c7598697998510ade3aad60a27880ee0e0003cf14bc22c6a166e49f9676"} Oct 31 19:45:01.173686 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:01.173374 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerStarted","Data":"5fcdd46f1b030fbc8f02294c67dcb67d60d160895ca7fb1ba10de9a4ace96bca"} Oct 31 19:45:01.288277 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:01.288225 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" podStartSLOduration=1.2882111840000001 podStartE2EDuration="1.288211184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:01.286795098 +0000 UTC m=+5021.888936242" watchObservedRunningTime="2026-10-31 19:45:01.288211184 +0000 UTC m=+5021.890352322" Oct 31 19:45:02.180016 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:02.179978 5688 generic.go:356] "Generic (PLEG): container finished" podID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerID="5f137c7598697998510ade3aad60a27880ee0e0003cf14bc22c6a166e49f9676" exitCode=60 Oct 31 19:45:02.180427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:02.180051 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerDied","Data":"5f137c7598697998510ade3aad60a27880ee0e0003cf14bc22c6a166e49f9676"} Oct 31 19:45:02.180427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:02.180345 5688 scope.go:122] "RemoveContainer" containerID="5f137c7598697998510ade3aad60a27880ee0e0003cf14bc22c6a166e49f9676" Oct 31 19:45:03.187204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:03.187102 5688 generic.go:356] "Generic (PLEG): container finished" podID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerID="69b2c24a5eb5015b2d3c83118394f2b188213ec4b5c3152996a7de4bc395da38" exitCode=60 Oct 31 19:45:03.187566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:03.187212 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerDied","Data":"69b2c24a5eb5015b2d3c83118394f2b188213ec4b5c3152996a7de4bc395da38"} Oct 31 19:45:03.187566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:03.187288 5688 scope.go:122] "RemoveContainer" containerID="5f137c7598697998510ade3aad60a27880ee0e0003cf14bc22c6a166e49f9676" Oct 31 19:45:03.187566 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:03.187374 5688 scope.go:122] "RemoveContainer" containerID="69b2c24a5eb5015b2d3c83118394f2b188213ec4b5c3152996a7de4bc395da38" Oct 31 19:45:03.187685 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:45:03.187573 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75_odh-ai-gateway-infra(e9cb081a-061d-4dde-b191-390175ff3d4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" Oct 31 19:45:04.194445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:04.194418 5688 scope.go:122] "RemoveContainer" containerID="69b2c24a5eb5015b2d3c83118394f2b188213ec4b5c3152996a7de4bc395da38" Oct 31 19:45:04.194820 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:45:04.194647 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75_odh-ai-gateway-infra(e9cb081a-061d-4dde-b191-390175ff3d4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" Oct 31 19:45:15.560121 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:15.559779 5688 scope.go:122] "RemoveContainer" containerID="69b2c24a5eb5015b2d3c83118394f2b188213ec4b5c3152996a7de4bc395da38" Oct 31 19:45:16.260076 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:16.260044 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerStarted","Data":"140570193f715cb41fa9cbc44c7c0ed7c440bf1947d36f7fcbd393006b08a241"} Oct 31 19:45:17.267467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:17.267435 5688 generic.go:356] "Generic (PLEG): container finished" podID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerID="140570193f715cb41fa9cbc44c7c0ed7c440bf1947d36f7fcbd393006b08a241" exitCode=60 Oct 31 19:45:17.267956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:17.267513 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerDied","Data":"140570193f715cb41fa9cbc44c7c0ed7c440bf1947d36f7fcbd393006b08a241"} Oct 31 19:45:17.267956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:17.267588 5688 scope.go:122] "RemoveContainer" containerID="69b2c24a5eb5015b2d3c83118394f2b188213ec4b5c3152996a7de4bc395da38" Oct 31 19:45:17.267956 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:17.267840 5688 scope.go:122] "RemoveContainer" containerID="140570193f715cb41fa9cbc44c7c0ed7c440bf1947d36f7fcbd393006b08a241" Oct 31 19:45:17.268098 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:45:17.268036 5688 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75_odh-ai-gateway-infra(e9cb081a-061d-4dde-b191-390175ff3d4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" Oct 31 19:45:17.404341 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:17.404316 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75"] Oct 31 19:45:18.410825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.410799 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:18.535646 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.535587 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca\") pod \"e9cb081a-061d-4dde-b191-390175ff3d4f\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " Oct 31 19:45:18.535779 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.535658 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd\" (UniqueName: \"kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd\") pod \"e9cb081a-061d-4dde-b191-390175ff3d4f\" (UID: \"e9cb081a-061d-4dde-b191-390175ff3d4f\") " Oct 31 19:45:18.535920 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.535898 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca" pod "e9cb081a-061d-4dde-b191-390175ff3d4f" (UID: "e9cb081a-061d-4dde-b191-390175ff3d4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:18.538262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.538236 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd" pod "e9cb081a-061d-4dde-b191-390175ff3d4f" (UID: "e9cb081a-061d-4dde-b191-390175ff3d4f"). InnerVolumeSpecName "kube-api-access-jqvgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:18.636786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.636763 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb081a-061d-4dde-b191-390175ff3d4f-maas-api-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:45:18.636786 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:18.636787 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvgd\" (UniqueName: \"kubernetes.io/projected/e9cb081a-061d-4dde-b191-390175ff3d4f-kube-api-access-jqvgd\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:45:19.280277 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:19.280247 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" Oct 31 19:45:19.280476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:19.280285 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75" event={"ID":"e9cb081a-061d-4dde-b191-390175ff3d4f","Type":"ContainerDied","Data":"5fcdd46f1b030fbc8f02294c67dcb67d60d160895ca7fb1ba10de9a4ace96bca"} Oct 31 19:45:19.280476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:19.280346 5688 scope.go:122] "RemoveContainer" containerID="140570193f715cb41fa9cbc44c7c0ed7c440bf1947d36f7fcbd393006b08a241" Oct 31 19:45:19.444450 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:19.444423 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75"] Oct 31 19:45:19.493929 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:19.493897 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0f794f-29891265-8ss75"] Oct 31 19:45:19.565362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:19.565292 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" path="/var/lib/kubelet/pods/e9cb081a-061d-4dde-b191-390175ff3d4f/volumes" Oct 31 19:45:22.126109 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:22.126078 5688 scope.go:122] "RemoveContainer" containerID="9d2e941c31d9ee3499dfb5c74251f278cd126eb2b6af37f156420fbc27cf5f8c" Oct 31 19:45:22.134795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:22.134777 5688 scope.go:122] "RemoveContainer" containerID="bd0812a86cc7ad68388aaa72ddfc03ce87588308dc1cbcd1668e06b322d20598" Oct 31 19:45:22.142298 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:22.142270 5688 scope.go:122] "RemoveContainer" containerID="4a8155457dc8af875d25c48df5071bf1eee3925ec0a178e92ccf1b47a3d85fa5" Oct 31 19:45:22.151067 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:22.151032 5688 scope.go:122] "RemoveContainer" containerID="a81ef0c4227b2e5d82b6abead8eb3f7d13ea70c40f423beb78711a163cf6fca0" Oct 31 19:45:22.158934 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:22.158918 5688 scope.go:122] "RemoveContainer" containerID="e2c0bb36fb3fbf09a9fe41099ce8c15a80f1ec7e724fa3e38918bf18a8b2fdaf" Oct 31 19:45:24.904310 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:45:24.904271 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:45:31.045274 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:31.045202 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth"] Oct 31 19:45:31.094657 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:31.094628 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a12e32-mllth"] Oct 31 19:45:31.565029 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:31.564996 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18309cf5-0f02-4e22-8abc-416b28542238" path="/var/lib/kubelet/pods/18309cf5-0f02-4e22-8abc-416b28542238/volumes" Oct 31 19:45:34.920701 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:34.920666 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g"] Oct 31 19:45:39.921762 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:39.921718 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" containerName="payload-processing" containerID="cri-o://21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046" gracePeriod=25 Oct 31 19:45:42.993105 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:42.993069 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:45:44.064812 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:44.064778 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx"] Oct 31 19:45:44.114226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:44.114195 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3833fc3a-4j6fx"] Oct 31 19:45:45.564897 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:45.564866 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04df1b12-1787-4992-9563-104c12b5e2ac" path="/var/lib/kubelet/pods/04df1b12-1787-4992-9563-104c12b5e2ac/volumes" Oct 31 19:45:52.992983 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:52.992950 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:45:54.903320 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:45:54.903283 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:45:57.502014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:57.501981 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g"] Oct 31 19:45:57.502454 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:45:57.502178 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" containerID="cri-o://74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882" gracePeriod=30 Oct 31 19:46:00.161229 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.161202 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:46:00.275690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.275666 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume\") pod \"0a5273ca-0444-433d-9814-0af5f263fe67\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " Oct 31 19:46:00.275841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.275774 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf\" (UniqueName: \"kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf\") pod \"0a5273ca-0444-433d-9814-0af5f263fe67\" (UID: \"0a5273ca-0444-433d-9814-0af5f263fe67\") " Oct 31 19:46:00.276337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.276111 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume" pod "0a5273ca-0444-433d-9814-0af5f263fe67" (UID: "0a5273ca-0444-433d-9814-0af5f263fe67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:46:00.278682 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.278656 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf" pod "0a5273ca-0444-433d-9814-0af5f263fe67" (UID: "0a5273ca-0444-433d-9814-0af5f263fe67"). InnerVolumeSpecName "kube-api-access-kfwqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:00.377108 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.377087 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5273ca-0444-433d-9814-0af5f263fe67-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:00.377252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.377109 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfwqf\" (UniqueName: \"kubernetes.io/projected/0a5273ca-0444-433d-9814-0af5f263fe67-kube-api-access-kfwqf\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:00.474994 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.474965 5688 generic.go:356] "Generic (PLEG): container finished" podID="0a5273ca-0444-433d-9814-0af5f263fe67" containerID="21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046" exitCode=0 Oct 31 19:46:00.475137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.475012 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" Oct 31 19:46:00.475137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.475025 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" event={"ID":"0a5273ca-0444-433d-9814-0af5f263fe67","Type":"ContainerDied","Data":"21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046"} Oct 31 19:46:00.475137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.475074 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g" event={"ID":"0a5273ca-0444-433d-9814-0af5f263fe67","Type":"ContainerDied","Data":"6bdf0c4fecdb370bde1732e986237289c48a8c0eaa73d10c66b183d23b213dfa"} Oct 31 19:46:00.475137 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.475096 5688 scope.go:122] "RemoveContainer" containerID="21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046" Oct 31 19:46:00.484541 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.484523 5688 scope.go:122] "RemoveContainer" containerID="21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046" Oct 31 19:46:00.484820 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:00.484796 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046\": container with ID starting with 21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046 not found: ID does not exist" containerID="21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046" Oct 31 19:46:00.484925 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.484828 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046"} err="failed to get container status \"21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046\": rpc error: code = NotFound desc = could not find container \"21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046\": container with ID starting with 21eec2d49f1045fb5f0a662297dee9545a7d70951459dec8608f75c86bf95046 not found: ID does not exist" Oct 31 19:46:00.641703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.641635 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g"] Oct 31 19:46:00.646406 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:00.646386 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0f794f-b4bcd995f-hxd8g"] Oct 31 19:46:01.564250 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:01.564221 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" path="/var/lib/kubelet/pods/0a5273ca-0444-433d-9814-0af5f263fe67/volumes" Oct 31 19:46:02.442382 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.442344 5688 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:46:02.442639 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.442395 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:46:02.748982 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.748959 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:46:02.799201 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799160 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799211 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799337 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799230 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799351 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799445 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799414 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799440 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:02.799518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799448 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799487 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:02.799518 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799508 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799546 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799597 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo\") pod \"087b9573-9e58-42ae-a315-80911d84e069\" (UID: \"087b9573-9e58-42ae-a315-80911d84e069\") " Oct 31 19:46:02.799712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799668 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:02.799712 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799704 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:02.799945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799713 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:46:02.799945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799894 5688 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087b9573-9e58-42ae-a315-80911d84e069-istiod-ca-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.799945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799911 5688 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-credential-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.799945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799921 5688 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.799945 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799937 5688 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-data\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.800234 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.799952 5688 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-workload-socket\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.801644 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.801618 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:02.801753 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.801685 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "kube-api-access-6ggb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:02.801753 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.801727 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:02.801876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.801779 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo" pod "087b9573-9e58-42ae-a315-80911d84e069" (UID: "087b9573-9e58-42ae-a315-80911d84e069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:46:02.901096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.901074 5688 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087b9573-9e58-42ae-a315-80911d84e069-istio-envoy\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.901096 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.901096 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggb7\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-kube-api-access-6ggb7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.901241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.901107 5688 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087b9573-9e58-42ae-a315-80911d84e069-istio-token\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:02.901241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:02.901115 5688 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087b9573-9e58-42ae-a315-80911d84e069-istio-podinfo\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:03.498387 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.498357 5688 generic.go:356] "Generic (PLEG): container finished" podID="087b9573-9e58-42ae-a315-80911d84e069" containerID="74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882" exitCode=0 Oct 31 19:46:03.498568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.498421 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" Oct 31 19:46:03.498568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.498431 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" event={"ID":"087b9573-9e58-42ae-a315-80911d84e069","Type":"ContainerDied","Data":"74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882"} Oct 31 19:46:03.498568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.498492 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g" event={"ID":"087b9573-9e58-42ae-a315-80911d84e069","Type":"ContainerDied","Data":"40b00adbe9ffdf6523c345dc5b21c26f0a863d7166dacb628a260ef7b16a831f"} Oct 31 19:46:03.498568 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.498513 5688 scope.go:122] "RemoveContainer" containerID="74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882" Oct 31 19:46:03.507987 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.507973 5688 scope.go:122] "RemoveContainer" containerID="978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62" Oct 31 19:46:03.514795 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.514772 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb"] Oct 31 19:46:03.518838 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.518766 5688 scope.go:122] "RemoveContainer" containerID="74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882" Oct 31 19:46:03.519257 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:03.519230 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882\": container with ID starting with 74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882 not found: ID does not exist" containerID="74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882" Oct 31 19:46:03.519357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.519265 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882"} err="failed to get container status \"74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882\": rpc error: code = NotFound desc = could not find container \"74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882\": container with ID starting with 74c0402946c76e813df7ecb02f1865aa4db5f24eb7437309e58df71a13de4882 not found: ID does not exist" Oct 31 19:46:03.519357 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.519282 5688 scope.go:122] "RemoveContainer" containerID="978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62" Oct 31 19:46:03.519580 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:03.519559 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62\": container with ID starting with 978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62 not found: ID does not exist" containerID="978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62" Oct 31 19:46:03.519633 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.519587 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62"} err="failed to get container status \"978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62\": rpc error: code = NotFound desc = could not find container \"978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62\": container with ID starting with 978acd9a41dde3cbdfdca52a087e94eb1237bcc5c3123684d0605353663a1e62 not found: ID does not exist" Oct 31 19:46:03.614204 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.614158 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr"] Oct 31 19:46:03.798071 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.798004 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g"] Oct 31 19:46:03.801730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:03.801705 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0f794f-openshift-default-69d6b8f98f-dx44g"] Oct 31 19:46:05.564036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:05.564003 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087b9573-9e58-42ae-a315-80911d84e069" path="/var/lib/kubelet/pods/087b9573-9e58-42ae-a315-80911d84e069/volumes" Oct 31 19:46:08.515448 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:08.515413 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerName="payload-processing" containerID="cri-o://78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab" gracePeriod=25 Oct 31 19:46:08.614556 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:08.614519 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" containerName="payload-pre-processing" containerID="cri-o://e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2" gracePeriod=25 Oct 31 19:46:17.030575 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:17.030532 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:46:17.031031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:17.030609 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:46:18.929933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:18.929898 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp"] Oct 31 19:46:18.979730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:18.979694 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms"] Oct 31 19:46:18.981013 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:18.980977 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv"] Oct 31 19:46:20.299454 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:20.299430 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:46:20.306001 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:20.305975 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:46:22.209448 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:22.209418 5688 scope.go:122] "RemoveContainer" containerID="697c9b3fa8153fb5ed0b28522b65dfbb068f204469f23da302fc5ccb582bfb0f" Oct 31 19:46:22.218245 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:22.218224 5688 scope.go:122] "RemoveContainer" containerID="2f08f59faefff069754ff2a9b65926acdbc54206ca1433511818c39e0fa1b48a" Oct 31 19:46:24.904328 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:24.904291 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:46:27.030392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:27.030356 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:46:27.030754 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:27.030417 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:46:28.758659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:28.758630 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:46:28.858144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:28.858119 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:46:28.912893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:28.912867 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume\") pod \"04ded44c-70e0-403b-878b-ed91a81a3f4e\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " Oct 31 19:46:28.912893 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:28.912897 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l\" (UniqueName: \"kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l\") pod \"04ded44c-70e0-403b-878b-ed91a81a3f4e\" (UID: \"04ded44c-70e0-403b-878b-ed91a81a3f4e\") " Oct 31 19:46:28.913226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:28.913178 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume" pod "04ded44c-70e0-403b-878b-ed91a81a3f4e" (UID: "04ded44c-70e0-403b-878b-ed91a81a3f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:46:28.915059 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:28.915040 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l" pod "04ded44c-70e0-403b-878b-ed91a81a3f4e" (UID: "04ded44c-70e0-403b-878b-ed91a81a3f4e"). InnerVolumeSpecName "kube-api-access-km22l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:29.014340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.014313 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume\") pod \"021b5d79-468d-4a37-a477-f358b903f7ab\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " Oct 31 19:46:29.014464 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.014427 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h\" (UniqueName: \"kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h\") pod \"021b5d79-468d-4a37-a477-f358b903f7ab\" (UID: \"021b5d79-468d-4a37-a477-f358b903f7ab\") " Oct 31 19:46:29.014676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.014660 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ded44c-70e0-403b-878b-ed91a81a3f4e-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:29.014744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.014680 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km22l\" (UniqueName: \"kubernetes.io/projected/04ded44c-70e0-403b-878b-ed91a81a3f4e-kube-api-access-km22l\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:29.014744 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.014674 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume" pod "021b5d79-468d-4a37-a477-f358b903f7ab" (UID: "021b5d79-468d-4a37-a477-f358b903f7ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:46:29.016467 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.016448 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h" pod "021b5d79-468d-4a37-a477-f358b903f7ab" (UID: "021b5d79-468d-4a37-a477-f358b903f7ab"). InnerVolumeSpecName "kube-api-access-vgk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:29.115437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.115380 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgk8h\" (UniqueName: \"kubernetes.io/projected/021b5d79-468d-4a37-a477-f358b903f7ab-kube-api-access-vgk8h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:29.115437 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.115401 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021b5d79-468d-4a37-a477-f358b903f7ab-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:29.624796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.624766 5688 generic.go:356] "Generic (PLEG): container finished" podID="021b5d79-468d-4a37-a477-f358b903f7ab" containerID="e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2" exitCode=0 Oct 31 19:46:29.624914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.624819 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" Oct 31 19:46:29.624914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.624845 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" event={"ID":"021b5d79-468d-4a37-a477-f358b903f7ab","Type":"ContainerDied","Data":"e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2"} Oct 31 19:46:29.624914 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.624896 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr" event={"ID":"021b5d79-468d-4a37-a477-f358b903f7ab","Type":"ContainerDied","Data":"d19ae46b5307ebd73b7ee280feaeae82977f83c87cc0be974fcb4ab0cb52d74e"} Oct 31 19:46:29.625077 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.624923 5688 scope.go:122] "RemoveContainer" containerID="e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2" Oct 31 19:46:29.626256 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.626235 5688 generic.go:356] "Generic (PLEG): container finished" podID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerID="78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab" exitCode=0 Oct 31 19:46:29.626392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.626303 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" Oct 31 19:46:29.626392 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.626352 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" event={"ID":"04ded44c-70e0-403b-878b-ed91a81a3f4e","Type":"ContainerDied","Data":"78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab"} Oct 31 19:46:29.626521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.626392 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb" event={"ID":"04ded44c-70e0-403b-878b-ed91a81a3f4e","Type":"ContainerDied","Data":"07074a3cd3a809d0c146504188318237c8a99379902b732202f1ecc004f560e5"} Oct 31 19:46:29.633870 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.633726 5688 scope.go:122] "RemoveContainer" containerID="e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2" Oct 31 19:46:29.634006 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:29.633989 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2\": container with ID starting with e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2 not found: ID does not exist" containerID="e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2" Oct 31 19:46:29.634051 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.634012 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2"} err="failed to get container status \"e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2\": rpc error: code = NotFound desc = could not find container \"e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2\": container with ID starting with e8bbf5eaf66fbb348b3a26e743bb3ba6a13214d91b99a8614c756de0061bd6d2 not found: ID does not exist" Oct 31 19:46:29.634051 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.634030 5688 scope.go:122] "RemoveContainer" containerID="78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab" Oct 31 19:46:29.641962 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.641949 5688 scope.go:122] "RemoveContainer" containerID="78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab" Oct 31 19:46:29.642199 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:29.642163 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab\": container with ID starting with 78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab not found: ID does not exist" containerID="78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab" Oct 31 19:46:29.642262 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.642208 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab"} err="failed to get container status \"78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab\": rpc error: code = NotFound desc = could not find container \"78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab\": container with ID starting with 78346b89aa4e1068f09cc36a86ecb76460bd59e6fd74fd503d9bd4298d09a4ab not found: ID does not exist" Oct 31 19:46:29.789476 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.789412 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb"] Oct 31 19:46:29.838896 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.838869 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f0e4e-87bd99997-l9ssb"] Oct 31 19:46:29.909129 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.909106 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr"] Oct 31 19:46:29.915000 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:29.914981 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f0e4e-6d7f66656c-fnhfr"] Oct 31 19:46:31.564804 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:31.564772 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" path="/var/lib/kubelet/pods/021b5d79-468d-4a37-a477-f358b903f7ab/volumes" Oct 31 19:46:31.565237 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:31.565221 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" path="/var/lib/kubelet/pods/04ded44c-70e0-403b-878b-ed91a81a3f4e/volumes" Oct 31 19:46:33.949584 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:33.949522 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" containerID="cri-o://d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f" gracePeriod=45 Oct 31 19:46:33.998719 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:33.998687 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" containerID="cri-o://77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031" gracePeriod=45 Oct 31 19:46:34.001339 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.001306 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" containerID="cri-o://bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970" gracePeriod=45 Oct 31 19:46:34.192152 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.192114 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Oct 31 19:46:34.246900 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.246874 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 19:46:34.265676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.265655 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 19:46:34.284328 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.284309 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 19:46:34.354173 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354147 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home\") pod \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " Oct 31 19:46:34.354338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354201 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache\") pod \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " Oct 31 19:46:34.354338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354238 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm\") pod \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " Oct 31 19:46:34.354338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354273 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir\") pod \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " Oct 31 19:46:34.354338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354300 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs\") pod \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " Oct 31 19:46:34.354338 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354337 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home\") pod \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354357 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache\") pod \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354381 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs\" (UniqueName: \"kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs\") pod \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354417 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq\" (UniqueName: \"kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq\") pod \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\" (UID: \"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14\") " Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354432 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home" pod "15c93d0a-85c2-4c9d-883b-fa4978bfa528" (UID: "15c93d0a-85c2-4c9d-883b-fa4978bfa528"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354444 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir\") pod \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354526 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp\" (UniqueName: \"kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp\") pod \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354526 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir" pod "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" (UID: "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.354621 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354548 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache" pod "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" (UID: "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354637 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir" pod "15c93d0a-85c2-4c9d-883b-fa4978bfa528" (UID: "15c93d0a-85c2-4c9d-883b-fa4978bfa528"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354675 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm\") pod \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\" (UID: \"15c93d0a-85c2-4c9d-883b-fa4978bfa528\") " Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.354888 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home" pod "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" (UID: "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.355010 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache" pod "15c93d0a-85c2-4c9d-883b-fa4978bfa528" (UID: "15c93d0a-85c2-4c9d-883b-fa4978bfa528"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.355026 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.355048 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.355063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.355062 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.355635 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.355077 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.355635 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.355090 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.357098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.357062 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs" pod "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" (UID: "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:46:34.357272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.357248 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm" pod "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" (UID: "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.357272 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.357254 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs" pod "15c93d0a-85c2-4c9d-883b-fa4978bfa528" (UID: "15c93d0a-85c2-4c9d-883b-fa4978bfa528"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:46:34.357471 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.357459 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp" pod "15c93d0a-85c2-4c9d-883b-fa4978bfa528" (UID: "15c93d0a-85c2-4c9d-883b-fa4978bfa528"). InnerVolumeSpecName "kube-api-access-2hvbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:34.357577 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.357559 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm" pod "15c93d0a-85c2-4c9d-883b-fa4978bfa528" (UID: "15c93d0a-85c2-4c9d-883b-fa4978bfa528"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.357637 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.357576 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq" pod "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" (UID: "1c52dce0-0ca5-4bf2-a7c6-a62efe325f14"). InnerVolumeSpecName "kube-api-access-wzhxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:34.456340 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456312 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs\") pod \"83b91cf0-7975-4c70-bc07-807330d477ef\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " Oct 31 19:46:34.456524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456366 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home\") pod \"83b91cf0-7975-4c70-bc07-807330d477ef\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " Oct 31 19:46:34.456524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456393 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst\" (UniqueName: \"kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst\") pod \"83b91cf0-7975-4c70-bc07-807330d477ef\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " Oct 31 19:46:34.456524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456413 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir\") pod \"83b91cf0-7975-4c70-bc07-807330d477ef\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " Oct 31 19:46:34.456524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456471 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache\") pod \"83b91cf0-7975-4c70-bc07-807330d477ef\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " Oct 31 19:46:34.456524 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456499 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm\") pod \"83b91cf0-7975-4c70-bc07-807330d477ef\" (UID: \"83b91cf0-7975-4c70-bc07-807330d477ef\") " Oct 31 19:46:34.456796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456690 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home" pod "83b91cf0-7975-4c70-bc07-807330d477ef" (UID: "83b91cf0-7975-4c70-bc07-807330d477ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.456796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456703 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir" pod "83b91cf0-7975-4c70-bc07-807330d477ef" (UID: "83b91cf0-7975-4c70-bc07-807330d477ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.456796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456695 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c93d0a-85c2-4c9d-883b-fa4978bfa528-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.456796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456746 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzhxq\" (UniqueName: \"kubernetes.io/projected/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-kube-api-access-wzhxq\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.456796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456794 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache" pod "83b91cf0-7975-4c70-bc07-807330d477ef" (UID: "83b91cf0-7975-4c70-bc07-807330d477ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.457014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456798 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hvbp\" (UniqueName: \"kubernetes.io/projected/15c93d0a-85c2-4c9d-883b-fa4978bfa528-kube-api-access-2hvbp\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.457014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456831 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.457014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456847 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.457014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456861 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.457014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.456875 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c93d0a-85c2-4c9d-883b-fa4978bfa528-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.458521 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.458498 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst" pod "83b91cf0-7975-4c70-bc07-807330d477ef" (UID: "83b91cf0-7975-4c70-bc07-807330d477ef"). InnerVolumeSpecName "kube-api-access-85bst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:46:34.458590 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.458546 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm" pod "83b91cf0-7975-4c70-bc07-807330d477ef" (UID: "83b91cf0-7975-4c70-bc07-807330d477ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:46:34.458640 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.458580 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs" pod "83b91cf0-7975-4c70-bc07-807330d477ef" (UID: "83b91cf0-7975-4c70-bc07-807330d477ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:46:34.557924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.557895 5688 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-dshm\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.557924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.557921 5688 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b91cf0-7975-4c70-bc07-807330d477ef-tls-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.557924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.557929 5688 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-home\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.558363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.557939 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85bst\" (UniqueName: \"kubernetes.io/projected/83b91cf0-7975-4c70-bc07-807330d477ef-kube-api-access-85bst\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.558363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.557949 5688 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-tmp-dir\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.558363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.557957 5688 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b91cf0-7975-4c70-bc07-807330d477ef-model-cache\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:46:34.651388 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.651355 5688 generic.go:356] "Generic (PLEG): container finished" podID="83b91cf0-7975-4c70-bc07-807330d477ef" containerID="77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031" exitCode=0 Oct 31 19:46:34.651571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.651413 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" Oct 31 19:46:34.651571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.651435 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" event={"ID":"83b91cf0-7975-4c70-bc07-807330d477ef","Type":"ContainerDied","Data":"77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031"} Oct 31 19:46:34.651571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.651474 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms" event={"ID":"83b91cf0-7975-4c70-bc07-807330d477ef","Type":"ContainerDied","Data":"b605c617347c5673efbeff6f73d826283b2365362d6df1caf88416348160ce60"} Oct 31 19:46:34.651571 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.651497 5688 scope.go:122] "RemoveContainer" containerID="77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031" Oct 31 19:46:34.652911 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.652885 5688 generic.go:356] "Generic (PLEG): container finished" podID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerID="d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f" exitCode=0 Oct 31 19:46:34.653027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.652981 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" Oct 31 19:46:34.653027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.652993 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" event={"ID":"15c93d0a-85c2-4c9d-883b-fa4978bfa528","Type":"ContainerDied","Data":"d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f"} Oct 31 19:46:34.653027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.653023 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp" event={"ID":"15c93d0a-85c2-4c9d-883b-fa4978bfa528","Type":"ContainerDied","Data":"46476c102c5a1462c120d239e304ce208f5c79948a848a7e7947be15c9cb3c1d"} Oct 31 19:46:34.654604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.654581 5688 generic.go:356] "Generic (PLEG): container finished" podID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerID="bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970" exitCode=0 Oct 31 19:46:34.654707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.654615 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" event={"ID":"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14","Type":"ContainerDied","Data":"bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970"} Oct 31 19:46:34.654707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.654644 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" event={"ID":"1c52dce0-0ca5-4bf2-a7c6-a62efe325f14","Type":"ContainerDied","Data":"74f4aea3ad559e16f6bd36a3d524d17b023b03291907e3090908142b97d3a38e"} Oct 31 19:46:34.654707 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.654643 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv" Oct 31 19:46:34.661729 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.661711 5688 scope.go:122] "RemoveContainer" containerID="77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031" Oct 31 19:46:34.662095 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:34.662067 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031\": container with ID starting with 77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031 not found: ID does not exist" containerID="77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031" Oct 31 19:46:34.662306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.662276 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031"} err="failed to get container status \"77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031\": rpc error: code = NotFound desc = could not find container \"77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031\": container with ID starting with 77940bba43860d6d99c526f953eaa9053544cfdf2cc83b1c49f6c37afaba7031 not found: ID does not exist" Oct 31 19:46:34.662421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.662308 5688 scope.go:122] "RemoveContainer" containerID="d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f" Oct 31 19:46:34.673144 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.673125 5688 scope.go:122] "RemoveContainer" containerID="d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f" Oct 31 19:46:34.673797 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:34.673770 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f\": container with ID starting with d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f not found: ID does not exist" containerID="d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f" Oct 31 19:46:34.673884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.673816 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f"} err="failed to get container status \"d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f\": rpc error: code = NotFound desc = could not find container \"d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f\": container with ID starting with d6e46061541fbb36b93f3c99be2f1cf682bd6ca8d24755e4f3b1d9420fb98c6f not found: ID does not exist" Oct 31 19:46:34.673884 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.673839 5688 scope.go:122] "RemoveContainer" containerID="bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970" Oct 31 19:46:34.681362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.681345 5688 scope.go:122] "RemoveContainer" containerID="bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970" Oct 31 19:46:34.681597 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:34.681579 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970\": container with ID starting with bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970 not found: ID does not exist" containerID="bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970" Oct 31 19:46:34.681655 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.681604 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970"} err="failed to get container status \"bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970\": rpc error: code = NotFound desc = could not find container \"bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970\": container with ID starting with bae88302b658540a4c55e1a968c2b7ccdcb4ab9352ca9c50e969f0ba57623970 not found: ID does not exist" Oct 31 19:46:34.816106 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.816050 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms"] Oct 31 19:46:34.820998 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.820978 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/premium-w1-da6ea7-kserve-75874d5dd7-smpms"] Oct 31 19:46:34.929933 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.929910 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv"] Oct 31 19:46:34.934294 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:34.934273 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/simulator-w1-da6ea7-kserve-76b6c46bdb-bnwkv"] Oct 31 19:46:35.047410 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:35.047388 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp"] Oct 31 19:46:35.053354 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:35.053334 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-w1-da6ea7-kserve-645f66cf4-266dp"] Oct 31 19:46:35.564616 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:35.564588 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" path="/var/lib/kubelet/pods/15c93d0a-85c2-4c9d-883b-fa4978bfa528/volumes" Oct 31 19:46:35.564991 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:35.564976 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" path="/var/lib/kubelet/pods/1c52dce0-0ca5-4bf2-a7c6-a62efe325f14/volumes" Oct 31 19:46:35.565351 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:35.565337 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" path="/var/lib/kubelet/pods/83b91cf0-7975-4c70-bc07-807330d477ef/volumes" Oct 31 19:46:47.056802 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:47.056773 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc"] Oct 31 19:46:47.104772 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:47.104742 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37e974e-8vbwc"] Oct 31 19:46:47.564237 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:47.564204 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde" path="/var/lib/kubelet/pods/bd61c01e-6e8c-4d6a-8d6c-c6f9d385cdde/volumes" Oct 31 19:46:54.903554 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:46:54.903514 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:46:59.053789 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:59.053715 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp"] Oct 31 19:46:59.104331 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:59.104294 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3170b0a-hjtgp"] Oct 31 19:46:59.574283 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:46:59.574250 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad994526-68cf-4e96-a8bc-963c2cc9a328" path="/var/lib/kubelet/pods/ad994526-68cf-4e96-a8bc-963c2cc9a328/volumes" Oct 31 19:47:01.338658 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:01.338622 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4ddzf_5c4762c4-658b-4a35-bfc5-b89a29bb2e11/kuadrant-console-plugin/0.log" Oct 31 19:47:04.619850 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:04.619824 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-v99xr_17f0ac62-e7aa-4dea-8bae-dede767972e6/manager/0.log" Oct 31 19:47:05.255727 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:05.255700 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fbfd7b7f-hx8bn_43418aaa-e973-435b-b950-f23e7d30e5a8/manager/0.log" Oct 31 19:47:05.735020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:05.734995 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cgxbd_c75c1ba8-2b81-4a05-bf20-22aea0bdc837/manager/0.log" Oct 31 19:47:06.043542 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:06.043476 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws"] Oct 31 19:47:06.092869 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:06.092834 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78fb452d-tcpws"] Oct 31 19:47:07.563921 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:07.563889 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd" path="/var/lib/kubelet/pods/10faf1a3-f32e-4f54-b1c3-db6aa8f7edcd/volumes" Oct 31 19:47:11.045676 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:11.045641 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r"] Oct 31 19:47:11.094988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:11.094958 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47884d9a-jlt7r"] Oct 31 19:47:11.564579 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:11.564547 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eebb54ac-af21-446f-9847-c5da68cf7e2f" path="/var/lib/kubelet/pods/eebb54ac-af21-446f-9847-c5da68cf7e2f/volumes" Oct 31 19:47:21.047056 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:21.047023 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c"] Oct 31 19:47:21.096435 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:21.096405 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44186b3-smd9c"] Oct 31 19:47:21.564650 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:21.564620 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f290f8d4-6993-4b36-a8f7-f16f9285fa9f" path="/var/lib/kubelet/pods/f290f8d4-6993-4b36-a8f7-f16f9285fa9f/volumes" Oct 31 19:47:22.247703 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:22.247674 5688 scope.go:122] "RemoveContainer" containerID="cd2d7d4e151a8d057bb3f4cd6132016a1c164a853cc91f25bfb6d4c92d7db588" Oct 31 19:47:22.256014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:22.255989 5688 scope.go:122] "RemoveContainer" containerID="0eda36d3f7561d54a7f1ece1523571763401ac7d2972b443bbe9b9444e815ac8" Oct 31 19:47:23.042757 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:23.042721 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq"] Oct 31 19:47:23.092286 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:23.092259 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9071f1-rrvkq"] Oct 31 19:47:23.564060 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:47:23.564029 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a601b4-a673-47c4-97cf-680866392d09" path="/var/lib/kubelet/pods/99a601b4-a673-47c4-97cf-680866392d09/volumes" Oct 31 19:47:24.903843 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:47:24.903811 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:47:54.903662 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:47:54.903624 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:48:16.260031 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.259991 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-hx8bn"] Oct 31 19:48:16.260611 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.260198 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" podUID="43418aaa-e973-435b-b950-f23e7d30e5a8" containerName="manager" containerID="cri-o://740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea" gracePeriod=30 Oct 31 19:48:16.508177 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.508153 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 19:48:16.617576 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.617521 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert\") pod \"43418aaa-e973-435b-b950-f23e7d30e5a8\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " Oct 31 19:48:16.617576 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.617570 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs\") pod \"43418aaa-e973-435b-b950-f23e7d30e5a8\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " Oct 31 19:48:16.617726 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.617612 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7\" (UniqueName: \"kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7\") pod \"43418aaa-e973-435b-b950-f23e7d30e5a8\" (UID: \"43418aaa-e973-435b-b950-f23e7d30e5a8\") " Oct 31 19:48:16.619696 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.619673 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs" pod "43418aaa-e973-435b-b950-f23e7d30e5a8" (UID: "43418aaa-e973-435b-b950-f23e7d30e5a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:48:16.619803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.619690 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert" pod "43418aaa-e973-435b-b950-f23e7d30e5a8" (UID: "43418aaa-e973-435b-b950-f23e7d30e5a8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:48:16.619803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.619739 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7" pod "43418aaa-e973-435b-b950-f23e7d30e5a8" (UID: "43418aaa-e973-435b-b950-f23e7d30e5a8"). InnerVolumeSpecName "kube-api-access-tfnl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:16.718306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.718271 5688 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-webhook-cert\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:48:16.718306 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.718301 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43418aaa-e973-435b-b950-f23e7d30e5a8-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:48:16.718497 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:16.718315 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnl7\" (UniqueName: \"kubernetes.io/projected/43418aaa-e973-435b-b950-f23e7d30e5a8-kube-api-access-tfnl7\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:48:17.103226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.103178 5688 generic.go:356] "Generic (PLEG): container finished" podID="43418aaa-e973-435b-b950-f23e7d30e5a8" containerID="740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea" exitCode=0 Oct 31 19:48:17.103396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.103241 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" Oct 31 19:48:17.103396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.103270 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" event={"ID":"43418aaa-e973-435b-b950-f23e7d30e5a8","Type":"ContainerDied","Data":"740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea"} Oct 31 19:48:17.103396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.103320 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-hx8bn" event={"ID":"43418aaa-e973-435b-b950-f23e7d30e5a8","Type":"ContainerDied","Data":"9e68c68b9bf7304fce48e717aea6a83fc059c9d74091ebb17b851fd1726b7e3d"} Oct 31 19:48:17.103396 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.103341 5688 scope.go:122] "RemoveContainer" containerID="740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea" Oct 31 19:48:17.113104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.113089 5688 scope.go:122] "RemoveContainer" containerID="740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea" Oct 31 19:48:17.113358 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:48:17.113341 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea\": container with ID starting with 740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea not found: ID does not exist" containerID="740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea" Oct 31 19:48:17.113412 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.113366 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea"} err="failed to get container status \"740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea\": rpc error: code = NotFound desc = could not find container \"740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea\": container with ID starting with 740358d94eea0cc47adf073a33bc3dfeaf48c10c25b123c93c9db78c793741ea not found: ID does not exist" Oct 31 19:48:17.266677 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.266651 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-hx8bn"] Oct 31 19:48:17.316805 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.316785 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-hx8bn"] Oct 31 19:48:17.570017 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:17.569978 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43418aaa-e973-435b-b950-f23e7d30e5a8" path="/var/lib/kubelet/pods/43418aaa-e973-435b-b950-f23e7d30e5a8/volumes" Oct 31 19:48:22.295779 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:22.295748 5688 scope.go:122] "RemoveContainer" containerID="8302cfa689af8354666e9f9722938a79ae414556ec5f87430c9a64368dfd2c08" Oct 31 19:48:22.303734 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:22.303707 5688 scope.go:122] "RemoveContainer" containerID="fccff50f3062efbe0e3c392ca233cb92c598c5ae0666a8c8962e2332e15f30ee" Oct 31 19:48:22.310694 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:22.310668 5688 scope.go:122] "RemoveContainer" containerID="b9828241eb0b9745109a5118531a9fbba996596c1cb3747d52e237f365470f6c" Oct 31 19:48:22.317728 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:22.317713 5688 scope.go:122] "RemoveContainer" containerID="de94e643d0579fdb7a580180161055d91f5ff942428bcdf5919b937087b9f17b" Oct 31 19:48:22.326395 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:22.326380 5688 scope.go:122] "RemoveContainer" containerID="827c46dde4f5c7582425e4abeca2a1a493a95576b947d55b5446f78d53198d6e" Oct 31 19:48:22.333527 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:22.333507 5688 scope.go:122] "RemoveContainer" containerID="00369669d9385fb320f3d5f4b9ab5e5f7c231cec3c3d24f3b755cdb056f4a889" Oct 31 19:48:24.903835 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:48:24.903796 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:48:36.502141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502109 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k"] Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502456 5688 cpu_manager.go:414] "containerMap: removing container" podUID="43418aaa-e973-435b-b950-f23e7d30e5a8" containerName="manager" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502468 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43418aaa-e973-435b-b950-f23e7d30e5a8" containerName="manager" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502481 5688 cpu_manager.go:414] "containerMap: removing container" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerName="payload-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502486 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerName="payload-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502501 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502506 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502513 5688 cpu_manager.go:414] "containerMap: removing container" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502517 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502523 5688 cpu_manager.go:414] "containerMap: removing container" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" containerName="payload-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502527 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" containerName="payload-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502533 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502537 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502544 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502548 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502560 5688 cpu_manager.go:414] "containerMap: removing container" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502564 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502571 5688 cpu_manager.go:414] "containerMap: removing container" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502576 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502584 5688 cpu_manager.go:414] "containerMap: removing container" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502588 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502598 5688 cpu_manager.go:414] "containerMap: removing container" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502602 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502608 5688 cpu_manager.go:414] "containerMap: removing container" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" containerName="payload-pre-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502613 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" containerName="payload-pre-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502661 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b91cf0-7975-4c70-bc07-807330d477ef" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502668 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="021b5d79-468d-4a37-a477-f358b903f7ab" containerName="payload-pre-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502675 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="43418aaa-e973-435b-b950-f23e7d30e5a8" containerName="manager" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502681 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ded44c-70e0-403b-878b-ed91a81a3f4e" containerName="payload-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502686 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e00ec3-6702-4512-9022-c7e604564478" containerName="istio-proxy" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502693 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502698 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c93d0a-85c2-4c9d-883b-fa4978bfa528" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502702 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5273ca-0444-433d-9814-0af5f263fe67" containerName="payload-processing" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502708 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502713 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cb081a-061d-4dde-b191-390175ff3d4f" containerName="cleanup" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502718 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502724 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c52dce0-0ca5-4bf2-a7c6-a62efe325f14" containerName="main" Oct 31 19:48:36.504469 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.502731 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="087b9573-9e58-42ae-a315-80911d84e069" containerName="istio-proxy" Oct 31 19:48:36.505704 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.505686 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.558044 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.558021 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nsw28\"" Oct 31 19:48:36.567924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.567896 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.567936 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568036 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.567959 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.568067 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568111 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.568090 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t82rx\" (UniqueName: \"kubernetes.io/projected/b1c9fc38-2e60-4b56-a5dd-a223912fc013-kube-api-access-t82rx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.568120 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568205 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.568162 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.568256 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.568327 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.568314 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.610363 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.610341 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k"] Oct 31 19:48:36.669122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669090 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669122 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669121 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669350 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669140 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669350 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669160 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669350 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669211 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669350 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669237 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669365 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669404 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669430 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t82rx\" (UniqueName: \"kubernetes.io/projected/b1c9fc38-2e60-4b56-a5dd-a223912fc013-kube-api-access-t82rx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669554 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669513 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669781 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669568 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669842 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669816 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.669902 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.669854 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.670084 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.670064 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.671598 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.671577 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.671697 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.671649 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.726679 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.726656 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c9fc38-2e60-4b56-a5dd-a223912fc013-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.727008 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.726991 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t82rx\" (UniqueName: \"kubernetes.io/projected/b1c9fc38-2e60-4b56-a5dd-a223912fc013-kube-api-access-t82rx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ztb8k\" (UID: \"b1c9fc38-2e60-4b56-a5dd-a223912fc013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:36.817636 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:36.817586 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:37.237082 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:37.237050 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k"] Oct 31 19:48:37.240253 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:48:37.240227 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c9fc38_2e60_4b56_a5dd_a223912fc013.slice/crio-baacb7cf3c23d866e40a85f7d6ee33b8096fce742751fe851e1f9b7aa00d21ac WatchSource:0}: Error finding container baacb7cf3c23d866e40a85f7d6ee33b8096fce742751fe851e1f9b7aa00d21ac: Status 404 returned error can't find the container with id baacb7cf3c23d866e40a85f7d6ee33b8096fce742751fe851e1f9b7aa00d21ac Oct 31 19:48:37.242141 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:37.242110 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:48:37.242241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:37.242173 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:48:37.242241 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:37.242215 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:48:38.187314 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:38.187281 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" event={"ID":"b1c9fc38-2e60-4b56-a5dd-a223912fc013","Type":"ContainerStarted","Data":"f2e88d65d3c3918304fb6fca4be1c0ae065230b84c2dc8755617be7b0d302707"} Oct 31 19:48:38.187825 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:38.187332 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" event={"ID":"b1c9fc38-2e60-4b56-a5dd-a223912fc013","Type":"ContainerStarted","Data":"baacb7cf3c23d866e40a85f7d6ee33b8096fce742751fe851e1f9b7aa00d21ac"} Oct 31 19:48:38.817872 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:38.817837 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:38.819098 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:38.819075 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 19:48:38.819252 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:38.819114 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 19:48:39.818818 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:39.818786 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 19:48:39.819264 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:39.818834 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 19:48:40.818175 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:40.818140 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 19:48:40.818380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:40.818227 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 19:48:41.818427 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:41.818394 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 19:48:41.818854 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:41.818447 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 19:48:42.817913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:42.817873 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 19:48:42.818097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:42.817948 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 19:48:43.817963 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:43.817925 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 19:48:43.818384 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:43.817988 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 19:48:45.164014 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:45.163981 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:45.164504 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:45.164407 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:45.165375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:45.165355 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" Oct 31 19:48:45.331227 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:48:45.331149 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podStartSLOduration=9.33113432 podStartE2EDuration="9.33113432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:38.323810617 +0000 UTC m=+5238.925951762" watchObservedRunningTime="2026-10-31 19:48:45.33113432 +0000 UTC m=+5245.933275465" Oct 31 19:48:54.904038 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:48:54.904005 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:49:24.902623 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:49:24.902589 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:49:54.903918 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:49:54.903881 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:50:22.320835 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.320799 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx"] Oct 31 19:50:22.324356 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.324337 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.369988 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.369963 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p"] Oct 31 19:50:22.371730 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.371702 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.371851 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.371770 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3affcc65-df38-4880-8581-75f2dffc6b50-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.371913 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.371872 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.371971 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.371918 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3affcc65-df38-4880-8581-75f2dffc6b50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.372027 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.371997 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmmc\" (UniqueName: \"kubernetes.io/projected/3affcc65-df38-4880-8581-75f2dffc6b50-kube-api-access-8tmmc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.372087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.372034 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3affcc65-df38-4880-8581-75f2dffc6b50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.372087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.372065 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.372218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.372166 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.372288 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.372247 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.373428 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.373408 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.432362 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.432338 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx"] Oct 31 19:50:22.473319 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473291 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473332 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3affcc65-df38-4880-8581-75f2dffc6b50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473365 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2sk\" (UniqueName: \"kubernetes.io/projected/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-kube-api-access-tj2sk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.473465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473399 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.473465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473433 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.473465 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473457 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmmc\" (UniqueName: \"kubernetes.io/projected/3affcc65-df38-4880-8581-75f2dffc6b50-kube-api-access-8tmmc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473556 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3affcc65-df38-4880-8581-75f2dffc6b50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473597 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473635 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473660 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473695 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473712 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473727 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.473777 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473766 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.474211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473825 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.474211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473856 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.474211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473928 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.474211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.473978 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.474211 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.474106 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3affcc65-df38-4880-8581-75f2dffc6b50-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.475210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.474999 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3affcc65-df38-4880-8581-75f2dffc6b50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.475814 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.475543 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.476238 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.475996 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.479094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.478919 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3affcc65-df38-4880-8581-75f2dffc6b50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.479094 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.479052 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.482344 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.482323 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3affcc65-df38-4880-8581-75f2dffc6b50-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.529301 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.529274 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p"] Oct 31 19:50:22.538218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.535315 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmmc\" (UniqueName: \"kubernetes.io/projected/3affcc65-df38-4880-8581-75f2dffc6b50-kube-api-access-8tmmc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.538218 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.536673 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3affcc65-df38-4880-8581-75f2dffc6b50-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-85vdx\" (UID: \"3affcc65-df38-4880-8581-75f2dffc6b50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.575207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575109 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575207 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575163 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575230 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575300 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575337 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2sk\" (UniqueName: \"kubernetes.io/projected/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-kube-api-access-tj2sk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575367 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575422 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575400 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575439 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575473 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.575732 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575576 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.576020 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.575997 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.576150 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.576131 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.576321 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.576300 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.577104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.577058 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.579821 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.578668 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.580974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.580125 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.632813 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.632785 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2sk\" (UniqueName: \"kubernetes.io/projected/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-kube-api-access-tj2sk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.632924 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.632811 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3982da-e2ec-4ffb-9b3f-a805c03bcbab-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp75p\" (UID: \"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:22.637796 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.637767 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:22.690293 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:22.689385 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:23.077855 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.077821 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx"] Oct 31 19:50:23.117564 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:50:23.117537 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3affcc65_df38_4880_8581_75f2dffc6b50.slice/crio-3d1b6c4a43a7a54c4beac52ed3d6ff5b3db2c87da9643e9e2e7226924f6fef0f WatchSource:0}: Error finding container 3d1b6c4a43a7a54c4beac52ed3d6ff5b3db2c87da9643e9e2e7226924f6fef0f: Status 404 returned error can't find the container with id 3d1b6c4a43a7a54c4beac52ed3d6ff5b3db2c87da9643e9e2e7226924f6fef0f Oct 31 19:50:23.119581 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.119546 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:50:23.119708 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.119629 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:50:23.119708 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.119679 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:50:23.153841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.153757 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p"] Oct 31 19:50:23.157059 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:50:23.157036 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3982da_e2ec_4ffb_9b3f_a805c03bcbab.slice/crio-db7afdf3cb4a5a4d498f1a87c7be462ea78592eb7fb6072e410bb0002a0a893f WatchSource:0}: Error finding container db7afdf3cb4a5a4d498f1a87c7be462ea78592eb7fb6072e410bb0002a0a893f: Status 404 returned error can't find the container with id db7afdf3cb4a5a4d498f1a87c7be462ea78592eb7fb6072e410bb0002a0a893f Oct 31 19:50:23.159106 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.159078 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:50:23.159171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.159138 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:50:23.159171 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.159165 5688 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 19:50:23.618622 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.618572 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" event={"ID":"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab","Type":"ContainerStarted","Data":"fe42de53a8c91263591cfa8a7f4fe1c1b4de508255c6958958151fcd3c8a16a8"} Oct 31 19:50:23.619132 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.618637 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" event={"ID":"ea3982da-e2ec-4ffb-9b3f-a805c03bcbab","Type":"ContainerStarted","Data":"db7afdf3cb4a5a4d498f1a87c7be462ea78592eb7fb6072e410bb0002a0a893f"} Oct 31 19:50:23.620104 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.620079 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" event={"ID":"3affcc65-df38-4880-8581-75f2dffc6b50","Type":"ContainerStarted","Data":"32446cedbda7763a26b0924a5247a385783ac8fce2a3153a2faa17285c4e3dd3"} Oct 31 19:50:23.620226 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.620119 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" event={"ID":"3affcc65-df38-4880-8581-75f2dffc6b50","Type":"ContainerStarted","Data":"3d1b6c4a43a7a54c4beac52ed3d6ff5b3db2c87da9643e9e2e7226924f6fef0f"} Oct 31 19:50:23.656210 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.652278 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:23.690348 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.690322 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:23.872269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.870293 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podStartSLOduration=1.8702745090000001 podStartE2EDuration="1.870274509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:23.869644769 +0000 UTC m=+5344.471785914" watchObservedRunningTime="2026-10-31 19:50:23.870274509 +0000 UTC m=+5344.472415654" Oct 31 19:50:23.872269 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:23.870874 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podStartSLOduration=1.87085977 podStartE2EDuration="1.87085977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:23.751172915 +0000 UTC m=+5344.353314065" watchObservedRunningTime="2026-10-31 19:50:23.87085977 +0000 UTC m=+5344.473000916" Oct 31 19:50:24.650810 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:24.650776 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:50:24.651386 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:24.651357 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:50:24.690951 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:24.690925 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:50:24.691145 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:24.691124 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:50:24.906477 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:50:24.906395 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:50:25.638570 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:25.638535 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:50:25.638846 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:25.638818 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:50:25.689841 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:25.689807 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:50:25.690282 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:25.689874 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:50:26.640619 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:26.640582 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:50:26.641214 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:26.640852 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:50:26.689866 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:26.689830 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:50:26.690284 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:26.690256 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:50:27.672160 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:27.672127 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:50:27.672486 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:27.672455 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:50:27.689803 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:27.689774 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:50:27.689961 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:27.689912 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:50:28.641537 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:28.641501 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:50:28.641829 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:28.641799 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:50:28.690013 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:28.689984 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:50:28.690526 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:28.690502 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:50:29.638406 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:29.638377 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:50:29.638580 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:29.638420 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:50:29.690604 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:29.690570 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:50:29.690979 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:29.690622 5688 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:50:31.109076 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:31.109048 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:31.109656 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:31.109395 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:31.110333 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:31.110311 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" Oct 31 19:50:31.141784 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:31.141766 5688 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:31.142087 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:31.142058 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:31.142688 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:50:31.142670 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" Oct 31 19:50:54.904166 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:50:54.904121 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:51:20.327063 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:51:20.327034 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:51:20.334070 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:51:20.334047 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:51:24.904169 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:51:24.904128 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:51:54.904043 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:51:54.904006 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:52:24.903251 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:52:24.903218 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:52:27.244617 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.244584 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss"] Oct 31 19:52:27.248005 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.247989 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.275660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.275634 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.275780 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.275669 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqlt\" (UniqueName: \"kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.300367 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.300344 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e15e5\"" Oct 31 19:52:27.350021 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.349991 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2df17b12-dockercfg-fr4w8\"" Oct 31 19:52:27.351368 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.351345 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss"] Oct 31 19:52:27.376276 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.376246 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.376414 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.376285 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqlt\" (UniqueName: \"kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.376974 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.376953 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.401368 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.401340 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7"] Oct 31 19:52:27.404725 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.404711 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.433093 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.433073 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqlt\" (UniqueName: \"kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.453131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.453106 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7"] Oct 31 19:52:27.477380 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.477350 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tcl\" (UniqueName: \"kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.477502 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.477450 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.560307 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.560253 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:27.578737 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.578717 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.578864 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.578798 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tcl\" (UniqueName: \"kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.579375 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.579354 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.636043 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.635993 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tcl\" (UniqueName: \"kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.717176 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.717148 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:27.928569 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:27.928496 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss"] Oct 31 19:52:27.931270 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:52:27.931236 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169d9c02_d981_4743_b82d_d8f6f9263203.slice/crio-9a50997dc452cd3ae2ef0b1219b410a55dc5babefe8b18854e49b65a1df51c80 WatchSource:0}: Error finding container 9a50997dc452cd3ae2ef0b1219b410a55dc5babefe8b18854e49b65a1df51c80: Status 404 returned error can't find the container with id 9a50997dc452cd3ae2ef0b1219b410a55dc5babefe8b18854e49b65a1df51c80 Oct 31 19:52:28.086157 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.086125 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7"] Oct 31 19:52:28.088718 ip-10-0-140-47 kubenswrapper[5688]: W1031 19:52:28.088694 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd33d5a_63e6_4243_90a6_943805fd832c.slice/crio-330e1a1fff2530b645a0fa1367c4673e0b20c4749e9ecb97a2d86793fb903f50 WatchSource:0}: Error finding container 330e1a1fff2530b645a0fa1367c4673e0b20c4749e9ecb97a2d86793fb903f50: Status 404 returned error can't find the container with id 330e1a1fff2530b645a0fa1367c4673e0b20c4749e9ecb97a2d86793fb903f50 Oct 31 19:52:28.219508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.219450 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" event={"ID":"169d9c02-d981-4743-b82d-d8f6f9263203","Type":"ContainerStarted","Data":"5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5"} Oct 31 19:52:28.219508 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.219498 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" event={"ID":"169d9c02-d981-4743-b82d-d8f6f9263203","Type":"ContainerStarted","Data":"9a50997dc452cd3ae2ef0b1219b410a55dc5babefe8b18854e49b65a1df51c80"} Oct 31 19:52:28.219690 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.219557 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:28.221009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.220987 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" event={"ID":"dfd33d5a-63e6-4243-90a6-943805fd832c","Type":"ContainerStarted","Data":"bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7"} Oct 31 19:52:28.221116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.221029 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" event={"ID":"dfd33d5a-63e6-4243-90a6-943805fd832c","Type":"ContainerStarted","Data":"330e1a1fff2530b645a0fa1367c4673e0b20c4749e9ecb97a2d86793fb903f50"} Oct 31 19:52:28.221116 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.221061 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:28.337320 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.337269 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" podStartSLOduration=1.33725398 podStartE2EDuration="1.33725398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:28.335529906 +0000 UTC m=+5468.937671052" watchObservedRunningTime="2026-10-31 19:52:28.33725398 +0000 UTC m=+5468.939395125" Oct 31 19:52:28.466782 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:28.465296 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" podStartSLOduration=1.465276931 podStartE2EDuration="1.465276931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:28.464043707 +0000 UTC m=+5469.066184864" watchObservedRunningTime="2026-10-31 19:52:28.465276931 +0000 UTC m=+5469.067418077" Oct 31 19:52:39.228403 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:39.228377 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:52:39.228814 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:52:39.228437 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:52:54.903998 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:52:54.903958 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:53:24.902806 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:53:24.902727 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:53:54.903852 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:53:54.903817 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:53:55.712131 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:53:55.712102 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7"] Oct 31 19:53:55.823304 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:53:55.823276 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss"] Oct 31 19:54:00.712875 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:00.712827 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerName="payload-processing" containerID="cri-o://bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7" gracePeriod=25 Oct 31 19:54:00.823781 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:00.823746 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" containerName="payload-pre-processing" containerID="cri-o://5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5" gracePeriod=25 Oct 31 19:54:09.227275 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:09.227240 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:54:09.227674 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:09.227238 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:54:19.226441 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:19.226407 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:54:19.226876 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:19.226412 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:54:20.970447 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:20.970381 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:54:21.062413 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.062386 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:54:21.088910 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.088876 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume\") pod \"dfd33d5a-63e6-4243-90a6-943805fd832c\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " Oct 31 19:54:21.089170 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.089138 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl\" (UniqueName: \"kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl\") pod \"dfd33d5a-63e6-4243-90a6-943805fd832c\" (UID: \"dfd33d5a-63e6-4243-90a6-943805fd832c\") " Oct 31 19:54:21.089324 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.089272 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume" pod "dfd33d5a-63e6-4243-90a6-943805fd832c" (UID: "dfd33d5a-63e6-4243-90a6-943805fd832c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:21.089470 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.089452 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd33d5a-63e6-4243-90a6-943805fd832c-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:54:21.091562 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.091535 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl" pod "dfd33d5a-63e6-4243-90a6-943805fd832c" (UID: "dfd33d5a-63e6-4243-90a6-943805fd832c"). InnerVolumeSpecName "kube-api-access-b5tcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:21.190573 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.190544 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt\" (UniqueName: \"kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt\") pod \"169d9c02-d981-4743-b82d-d8f6f9263203\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " Oct 31 19:54:21.190720 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.190638 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume\") pod \"169d9c02-d981-4743-b82d-d8f6f9263203\" (UID: \"169d9c02-d981-4743-b82d-d8f6f9263203\") " Oct 31 19:54:21.190800 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.190789 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5tcl\" (UniqueName: \"kubernetes.io/projected/dfd33d5a-63e6-4243-90a6-943805fd832c-kube-api-access-b5tcl\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:54:21.190997 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.190977 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume" pod "169d9c02-d981-4743-b82d-d8f6f9263203" (UID: "169d9c02-d981-4743-b82d-d8f6f9263203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:21.192681 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.192664 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt" pod "169d9c02-d981-4743-b82d-d8f6f9263203" (UID: "169d9c02-d981-4743-b82d-d8f6f9263203"). InnerVolumeSpecName "kube-api-access-qrqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:21.291515 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.291492 5688 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169d9c02-d981-4743-b82d-d8f6f9263203-plugins-config-volume\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:54:21.291659 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.291519 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqlt\" (UniqueName: \"kubernetes.io/projected/169d9c02-d981-4743-b82d-d8f6f9263203-kube-api-access-qrqlt\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 19:54:21.702325 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.702301 5688 generic.go:356] "Generic (PLEG): container finished" podID="169d9c02-d981-4743-b82d-d8f6f9263203" containerID="5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5" exitCode=0 Oct 31 19:54:21.702457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.702349 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" Oct 31 19:54:21.702457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.702368 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" event={"ID":"169d9c02-d981-4743-b82d-d8f6f9263203","Type":"ContainerDied","Data":"5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5"} Oct 31 19:54:21.702457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.702407 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss" event={"ID":"169d9c02-d981-4743-b82d-d8f6f9263203","Type":"ContainerDied","Data":"9a50997dc452cd3ae2ef0b1219b410a55dc5babefe8b18854e49b65a1df51c80"} Oct 31 19:54:21.702457 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.702434 5688 scope.go:122] "RemoveContainer" containerID="5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5" Oct 31 19:54:21.703660 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.703637 5688 generic.go:356] "Generic (PLEG): container finished" podID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerID="bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7" exitCode=0 Oct 31 19:54:21.703801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.703667 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" event={"ID":"dfd33d5a-63e6-4243-90a6-943805fd832c","Type":"ContainerDied","Data":"bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7"} Oct 31 19:54:21.703801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.703691 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" Oct 31 19:54:21.703801 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.703702 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7" event={"ID":"dfd33d5a-63e6-4243-90a6-943805fd832c","Type":"ContainerDied","Data":"330e1a1fff2530b645a0fa1367c4673e0b20c4749e9ecb97a2d86793fb903f50"} Oct 31 19:54:21.710822 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.710803 5688 scope.go:122] "RemoveContainer" containerID="5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5" Oct 31 19:54:21.711056 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:54:21.711040 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5\": container with ID starting with 5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5 not found: ID does not exist" containerID="5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5" Oct 31 19:54:21.711103 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.711063 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5"} err="failed to get container status \"5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5\": rpc error: code = NotFound desc = could not find container \"5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5\": container with ID starting with 5017bcc33fc20317e1afd3f198b72ae192a7e2e25e517e1ceb2a8c78c5fa73e5 not found: ID does not exist" Oct 31 19:54:21.711103 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.711079 5688 scope.go:122] "RemoveContainer" containerID="bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7" Oct 31 19:54:21.718097 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.718080 5688 scope.go:122] "RemoveContainer" containerID="bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7" Oct 31 19:54:21.718367 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:54:21.718348 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7\": container with ID starting with bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7 not found: ID does not exist" containerID="bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7" Oct 31 19:54:21.718442 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.718379 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7"} err="failed to get container status \"bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7\": rpc error: code = NotFound desc = could not find container \"bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7\": container with ID starting with bf4768715d1a6fb4049fbc1ba4b3dcde1d4583fd7bef3e895b3d5f0a1f8286f7 not found: ID does not exist" Oct 31 19:54:21.867737 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.867667 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7"] Oct 31 19:54:21.875274 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:21.875251 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99226l7"] Oct 31 19:54:22.032089 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:22.032063 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss"] Oct 31 19:54:22.034499 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:22.034476 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d5sh9ss"] Oct 31 19:54:23.564009 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:23.563981 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" path="/var/lib/kubelet/pods/169d9c02-d981-4743-b82d-d8f6f9263203/volumes" Oct 31 19:54:23.564421 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:54:23.564341 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" path="/var/lib/kubelet/pods/dfd33d5a-63e6-4243-90a6-943805fd832c/volumes" Oct 31 19:54:24.904106 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:54:24.904075 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:54:54.902898 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:54:54.902864 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:55:24.906062 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:55:24.906028 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:55:54.902848 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:55:54.902814 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:56:20.353286 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:56:20.353203 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:56:20.360908 ip-10-0-140-47 kubenswrapper[5688]: I1031 19:56:20.360886 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 19:56:24.902740 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:56:24.902703 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:56:54.903052 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:56:54.903017 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:57:24.904085 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:57:24.904050 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:57:54.904067 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:57:54.904027 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:58:24.903367 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:58:24.903334 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:58:54.903056 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:58:54.903019 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:59:24.903495 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:59:24.903414 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 19:59:54.902783 ip-10-0-140-47 kubenswrapper[5688]: E1031 19:59:54.902749 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:00:24.902616 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:00:24.902577 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:00:54.903069 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:00:54.903033 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:01:20.379493 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:01:20.379464 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:01:20.388069 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:01:20.388049 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:01:24.903427 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:01:24.903392 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:01:54.903517 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:01:54.903476 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:02:02.153364 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:02:02.153273 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:02:02.153926 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:02:02.153892 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:02:02.187371 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:02:02.187331 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:02:02.187540 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:02:02.187385 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:02:24.904783 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:02:24.904706 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:02:54.928466 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:02:54.928429 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:03:24.903372 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:03:24.903335 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:03:47.157296 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:03:47.157257 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:03:47.163412 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:03:47.157823 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:03:47.188750 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:03:47.188722 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:03:47.188952 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:03:47.188928 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:03:54.903694 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:03:54.903658 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:04:24.902961 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:04:24.902922 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:04:54.903475 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:04:54.903440 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:05:24.903471 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:05:24.903393 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:05:54.904034 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:05:54.903999 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:06:20.404283 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:06:20.404259 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:06:20.423226 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:06:20.423175 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:06:24.903139 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:06:24.903102 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:06:54.922471 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:06:54.922399 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:07:24.904233 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:07:24.904145 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:07:54.905085 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:07:54.905047 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:08:24.904670 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:08:24.904591 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:08:54.904029 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:08:54.903990 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:09:24.903838 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:09:24.903800 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:09:54.905009 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:09:54.904971 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:10:12.243476 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:12.243439 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9645m"] Oct 31 20:10:12.244030 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:12.243632 5688 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" podUID="5b5750a4-d7e3-42fa-be66-f86555616beb" containerName="maas-api" containerID="cri-o://dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db" gracePeriod=30 Oct 31 20:10:13.183369 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.183322 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" podUID="5b5750a4-d7e3-42fa-be66-f86555616beb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.35:8443/health\": dial tcp 10.132.0.35:8443: connect: connection refused" Oct 31 20:10:13.511222 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.511171 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 20:10:13.515303 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.515276 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs\") pod \"5b5750a4-d7e3-42fa-be66-f86555616beb\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " Oct 31 20:10:13.515452 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.515363 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca\") pod \"5b5750a4-d7e3-42fa-be66-f86555616beb\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " Oct 31 20:10:13.515452 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.515419 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk\") pod \"5b5750a4-d7e3-42fa-be66-f86555616beb\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " Oct 31 20:10:13.515578 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.515530 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls\") pod \"5b5750a4-d7e3-42fa-be66-f86555616beb\" (UID: \"5b5750a4-d7e3-42fa-be66-f86555616beb\") " Oct 31 20:10:13.515802 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.515760 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca" pod "5b5750a4-d7e3-42fa-be66-f86555616beb" (UID: "5b5750a4-d7e3-42fa-be66-f86555616beb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:10:13.517811 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.517779 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs" pod "5b5750a4-d7e3-42fa-be66-f86555616beb" (UID: "5b5750a4-d7e3-42fa-be66-f86555616beb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:10:13.518314 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.518290 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk" pod "5b5750a4-d7e3-42fa-be66-f86555616beb" (UID: "5b5750a4-d7e3-42fa-be66-f86555616beb"). InnerVolumeSpecName "kube-api-access-8t9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:10:13.518781 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.518760 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls" pod "5b5750a4-d7e3-42fa-be66-f86555616beb" (UID: "5b5750a4-d7e3-42fa-be66-f86555616beb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:10:13.616957 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.616758 5688 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-metrics-certs\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:13.616957 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.616802 5688 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5750a4-d7e3-42fa-be66-f86555616beb-openshift-service-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:13.616957 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.616820 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/5b5750a4-d7e3-42fa-be66-f86555616beb-kube-api-access-8t9wk\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:13.616957 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:13.616837 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5750a4-d7e3-42fa-be66-f86555616beb-maas-api-tls\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:14.086267 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.086221 5688 generic.go:356] "Generic (PLEG): container finished" podID="5b5750a4-d7e3-42fa-be66-f86555616beb" containerID="dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db" exitCode=0 Oct 31 20:10:14.086267 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.086260 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" event={"ID":"5b5750a4-d7e3-42fa-be66-f86555616beb","Type":"ContainerDied","Data":"dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db"} Oct 31 20:10:14.086569 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.086306 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" event={"ID":"5b5750a4-d7e3-42fa-be66-f86555616beb","Type":"ContainerDied","Data":"9432f478b8a2ae9e3d7003da73492b9a4b1f73c614901192c50951d2eb744ff3"} Oct 31 20:10:14.086569 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.086326 5688 scope.go:122] "RemoveContainer" containerID="dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db" Oct 31 20:10:14.086569 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.086244 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9645m" Oct 31 20:10:14.102294 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.102266 5688 scope.go:122] "RemoveContainer" containerID="dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db" Oct 31 20:10:14.102650 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:10:14.102626 5688 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db\": container with ID starting with dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db not found: ID does not exist" containerID="dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db" Oct 31 20:10:14.102772 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.102660 5688 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db"} err="failed to get container status \"dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db\": rpc error: code = NotFound desc = could not find container \"dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db\": container with ID starting with dcc15d79c30b36735ab1693e05766a28c74266730abe82b2467f381ad6aac8db not found: ID does not exist" Oct 31 20:10:14.249565 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.249511 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9645m"] Oct 31 20:10:14.301430 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:14.301387 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9645m"] Oct 31 20:10:15.599212 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:15.596341 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5750a4-d7e3-42fa-be66-f86555616beb" path="/var/lib/kubelet/pods/5b5750a4-d7e3-42fa-be66-f86555616beb/volumes" Oct 31 20:10:20.048928 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.048889 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd"] Oct 31 20:10:20.049356 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049288 5688 cpu_manager.go:414] "containerMap: removing container" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerName="payload-processing" Oct 31 20:10:20.049356 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049300 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerName="payload-processing" Oct 31 20:10:20.049356 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049321 5688 cpu_manager.go:414] "containerMap: removing container" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" containerName="payload-pre-processing" Oct 31 20:10:20.049356 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049328 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" containerName="payload-pre-processing" Oct 31 20:10:20.049356 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049346 5688 cpu_manager.go:414] "containerMap: removing container" podUID="5b5750a4-d7e3-42fa-be66-f86555616beb" containerName="maas-api" Oct 31 20:10:20.049356 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049350 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5750a4-d7e3-42fa-be66-f86555616beb" containerName="maas-api" Oct 31 20:10:20.049561 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049413 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5750a4-d7e3-42fa-be66-f86555616beb" containerName="maas-api" Oct 31 20:10:20.049561 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049426 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd33d5a-63e6-4243-90a6-943805fd832c" containerName="payload-processing" Oct 31 20:10:20.049561 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.049433 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="169d9c02-d981-4743-b82d-d8f6f9263203" containerName="payload-pre-processing" Oct 31 20:10:20.053655 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.053639 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.103779 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.103735 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.103974 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.103800 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4lg\" (UniqueName: \"kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.103974 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.103932 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.163228 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.163164 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:10:20.163228 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.163202 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:10:20.163228 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.163211 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:10:20.163477 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.163240 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:10:20.164426 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.164405 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd"] Oct 31 20:10:20.205468 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.205439 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.205671 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.205495 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.205671 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.205538 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4lg\" (UniqueName: \"kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.206267 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.206239 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.208037 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.208017 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.263201 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.263137 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4lg\" (UniqueName: \"kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg\") pod \"maas-api-revoke-sub-436da9be-jwcqd\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.366453 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.366366 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:20.738910 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:20.738875 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd"] Oct 31 20:10:20.742463 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:10:20.742435 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dd699e_7f86_4f71_9ac8_916aa4919973.slice/crio-bea2585299105133c0a8c1b7bc9a293f8bb3360d070280c3fc950a4fef92768a WatchSource:0}: Error finding container bea2585299105133c0a8c1b7bc9a293f8bb3360d070280c3fc950a4fef92768a: Status 404 returned error can't find the container with id bea2585299105133c0a8c1b7bc9a293f8bb3360d070280c3fc950a4fef92768a Oct 31 20:10:21.128529 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:21.128483 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" event={"ID":"80dd699e-7f86-4f71-9ac8-916aa4919973","Type":"ContainerStarted","Data":"632cf9cb27bc55c28361951b46d326906466f14c46a4e41f0256e93546a3fd68"} Oct 31 20:10:21.128902 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:21.128549 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" event={"ID":"80dd699e-7f86-4f71-9ac8-916aa4919973","Type":"ContainerStarted","Data":"bea2585299105133c0a8c1b7bc9a293f8bb3360d070280c3fc950a4fef92768a"} Oct 31 20:10:21.247467 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:21.247412 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" podStartSLOduration=1.247394238 podStartE2EDuration="1.247394238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:10:21.245689811 +0000 UTC m=+6541.847830958" watchObservedRunningTime="2026-10-31 20:10:21.247394238 +0000 UTC m=+6541.849535380" Oct 31 20:10:22.134071 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:22.134033 5688 generic.go:356] "Generic (PLEG): container finished" podID="80dd699e-7f86-4f71-9ac8-916aa4919973" containerID="632cf9cb27bc55c28361951b46d326906466f14c46a4e41f0256e93546a3fd68" exitCode=0 Oct 31 20:10:22.134514 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:22.134125 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" event={"ID":"80dd699e-7f86-4f71-9ac8-916aa4919973","Type":"ContainerDied","Data":"632cf9cb27bc55c28361951b46d326906466f14c46a4e41f0256e93546a3fd68"} Oct 31 20:10:23.275259 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.275233 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:23.334884 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.334842 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca\") pod \"80dd699e-7f86-4f71-9ac8-916aa4919973\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " Oct 31 20:10:23.335074 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.334938 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca\") pod \"80dd699e-7f86-4f71-9ac8-916aa4919973\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " Oct 31 20:10:23.335074 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.334988 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg\" (UniqueName: \"kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg\") pod \"80dd699e-7f86-4f71-9ac8-916aa4919973\" (UID: \"80dd699e-7f86-4f71-9ac8-916aa4919973\") " Oct 31 20:10:23.335384 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.335350 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca" pod "80dd699e-7f86-4f71-9ac8-916aa4919973" (UID: "80dd699e-7f86-4f71-9ac8-916aa4919973"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:10:23.337135 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.337114 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca" pod "80dd699e-7f86-4f71-9ac8-916aa4919973" (UID: "80dd699e-7f86-4f71-9ac8-916aa4919973"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:10:23.337270 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.337250 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg" pod "80dd699e-7f86-4f71-9ac8-916aa4919973" (UID: "80dd699e-7f86-4f71-9ac8-916aa4919973"). InnerVolumeSpecName "kube-api-access-qn4lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:10:23.435642 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.435557 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:23.435642 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.435586 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4lg\" (UniqueName: \"kubernetes.io/projected/80dd699e-7f86-4f71-9ac8-916aa4919973-kube-api-access-qn4lg\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:23.435642 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:23.435601 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dd699e-7f86-4f71-9ac8-916aa4919973-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:24.144856 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:24.144819 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" event={"ID":"80dd699e-7f86-4f71-9ac8-916aa4919973","Type":"ContainerDied","Data":"bea2585299105133c0a8c1b7bc9a293f8bb3360d070280c3fc950a4fef92768a"} Oct 31 20:10:24.145068 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:24.144879 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bea2585299105133c0a8c1b7bc9a293f8bb3360d070280c3fc950a4fef92768a" Oct 31 20:10:24.145068 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:24.144968 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd" Oct 31 20:10:24.904066 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:10:24.904033 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:10:30.875155 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.875120 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78"] Oct 31 20:10:30.875622 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.875492 5688 cpu_manager.go:414] "containerMap: removing container" podUID="80dd699e-7f86-4f71-9ac8-916aa4919973" containerName="revoke-keys" Oct 31 20:10:30.875622 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.875503 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dd699e-7f86-4f71-9ac8-916aa4919973" containerName="revoke-keys" Oct 31 20:10:30.875622 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.875559 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dd699e-7f86-4f71-9ac8-916aa4919973" containerName="revoke-keys" Oct 31 20:10:30.881419 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.881400 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:30.936555 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.936524 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78"] Oct 31 20:10:30.984895 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.984857 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:10:30.985058 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.984898 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:10:30.985058 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.984933 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:10:30.985058 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:30.984992 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:10:31.005562 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.005537 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.005713 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.005572 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlsk8\" (UniqueName: \"kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.005713 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.005601 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.106944 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.106904 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlsk8\" (UniqueName: \"kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.107261 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.106960 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.107261 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.107054 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.107764 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.107740 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.109542 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.109519 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.165120 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.165044 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlsk8\" (UniqueName: \"kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8\") pod \"maas-api-revoke-sub-fdf94c23-j8k78\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.197916 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.197877 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:31.580247 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:31.580206 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78"] Oct 31 20:10:31.583292 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:10:31.583260 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6da421_b9ab_475f_a6b0_0cc4263598fd.slice/crio-ca40121667b542bd8e71d7147ed405cfdbb0f3913d148b7ca047fbdb58c0fd72 WatchSource:0}: Error finding container ca40121667b542bd8e71d7147ed405cfdbb0f3913d148b7ca047fbdb58c0fd72: Status 404 returned error can't find the container with id ca40121667b542bd8e71d7147ed405cfdbb0f3913d148b7ca047fbdb58c0fd72 Oct 31 20:10:32.179274 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:32.179170 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" event={"ID":"6b6da421-b9ab-475f-a6b0-0cc4263598fd","Type":"ContainerStarted","Data":"aacf0edac90a3a0fcffdf5c922a91159228e09a0ea1dc546d23e92a62dea83ff"} Oct 31 20:10:32.179274 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:32.179245 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" event={"ID":"6b6da421-b9ab-475f-a6b0-0cc4263598fd","Type":"ContainerStarted","Data":"ca40121667b542bd8e71d7147ed405cfdbb0f3913d148b7ca047fbdb58c0fd72"} Oct 31 20:10:32.294604 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:32.294554 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" podStartSLOduration=2.294535571 podStartE2EDuration="2.294535571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:10:32.29313943 +0000 UTC m=+6552.895280575" watchObservedRunningTime="2026-10-31 20:10:32.294535571 +0000 UTC m=+6552.896676716" Oct 31 20:10:33.185342 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:33.185309 5688 generic.go:356] "Generic (PLEG): container finished" podID="6b6da421-b9ab-475f-a6b0-0cc4263598fd" containerID="aacf0edac90a3a0fcffdf5c922a91159228e09a0ea1dc546d23e92a62dea83ff" exitCode=0 Oct 31 20:10:33.185796 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:33.185393 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" event={"ID":"6b6da421-b9ab-475f-a6b0-0cc4263598fd","Type":"ContainerDied","Data":"aacf0edac90a3a0fcffdf5c922a91159228e09a0ea1dc546d23e92a62dea83ff"} Oct 31 20:10:34.318492 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.318468 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:34.445630 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.445562 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca\") pod \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " Oct 31 20:10:34.445787 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.445701 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca\") pod \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " Oct 31 20:10:34.445787 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.445733 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8\" (UniqueName: \"kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8\") pod \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\" (UID: \"6b6da421-b9ab-475f-a6b0-0cc4263598fd\") " Oct 31 20:10:34.446073 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.446049 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca" pod "6b6da421-b9ab-475f-a6b0-0cc4263598fd" (UID: "6b6da421-b9ab-475f-a6b0-0cc4263598fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:10:34.447770 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.447747 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca" pod "6b6da421-b9ab-475f-a6b0-0cc4263598fd" (UID: "6b6da421-b9ab-475f-a6b0-0cc4263598fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:10:34.447901 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.447882 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8" pod "6b6da421-b9ab-475f-a6b0-0cc4263598fd" (UID: "6b6da421-b9ab-475f-a6b0-0cc4263598fd"). InnerVolumeSpecName "kube-api-access-zlsk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:10:34.547091 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.547063 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:34.547091 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.547087 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlsk8\" (UniqueName: \"kubernetes.io/projected/6b6da421-b9ab-475f-a6b0-0cc4263598fd-kube-api-access-zlsk8\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:34.547293 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:34.547098 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6da421-b9ab-475f-a6b0-0cc4263598fd-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:10:35.196171 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:35.196136 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" Oct 31 20:10:35.196358 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:35.196134 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78" event={"ID":"6b6da421-b9ab-475f-a6b0-0cc4263598fd","Type":"ContainerDied","Data":"ca40121667b542bd8e71d7147ed405cfdbb0f3913d148b7ca047fbdb58c0fd72"} Oct 31 20:10:35.196358 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:10:35.196291 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca40121667b542bd8e71d7147ed405cfdbb0f3913d148b7ca047fbdb58c0fd72" Oct 31 20:10:54.903745 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:10:54.903704 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:11:07.471353 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.471265 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs"] Oct 31 20:11:07.471754 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.471677 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6b6da421-b9ab-475f-a6b0-0cc4263598fd" containerName="revoke-keys" Oct 31 20:11:07.471754 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.471689 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6da421-b9ab-475f-a6b0-0cc4263598fd" containerName="revoke-keys" Oct 31 20:11:07.471827 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.471758 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6da421-b9ab-475f-a6b0-0cc4263598fd" containerName="revoke-keys" Oct 31 20:11:07.474909 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.474887 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.547806 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.547771 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.547806 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.547809 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d98h\" (UniqueName: \"kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.548035 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.547916 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.576944 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.576914 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:11:07.576944 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.576937 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:11:07.577154 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.576971 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:11:07.577154 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.577031 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:11:07.578483 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.578466 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs"] Oct 31 20:11:07.649287 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.649254 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.649505 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.649342 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.649505 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.649363 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d98h\" (UniqueName: \"kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.649967 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.649944 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.651809 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.651784 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.706769 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.706733 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d98h\" (UniqueName: \"kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h\") pod \"maas-api-revoke-sub-829b426d-d7lrs\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:07.787955 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:07.787921 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:08.161840 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:08.161676 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs"] Oct 31 20:11:08.164640 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:11:08.164612 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f87461_a7b7_4d97_b76c_b8053d252060.slice/crio-642880ec6d2b5c41136f31eced0b6ec23ac428ad618fb8a1af660f623588ceb8 WatchSource:0}: Error finding container 642880ec6d2b5c41136f31eced0b6ec23ac428ad618fb8a1af660f623588ceb8: Status 404 returned error can't find the container with id 642880ec6d2b5c41136f31eced0b6ec23ac428ad618fb8a1af660f623588ceb8 Oct 31 20:11:08.358690 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:08.358660 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" event={"ID":"a2f87461-a7b7-4d97-b76c-b8053d252060","Type":"ContainerStarted","Data":"642880ec6d2b5c41136f31eced0b6ec23ac428ad618fb8a1af660f623588ceb8"} Oct 31 20:11:09.364908 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:09.364824 5688 generic.go:356] "Generic (PLEG): container finished" podID="a2f87461-a7b7-4d97-b76c-b8053d252060" containerID="c48ae16d7da4470057ecd76fc943cb4d6519ae4384aa9c2b57f0bc266101fe05" exitCode=0 Oct 31 20:11:09.365321 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:09.364911 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" event={"ID":"a2f87461-a7b7-4d97-b76c-b8053d252060","Type":"ContainerDied","Data":"c48ae16d7da4470057ecd76fc943cb4d6519ae4384aa9c2b57f0bc266101fe05"} Oct 31 20:11:10.500224 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.500176 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:10.575138 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.575105 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca\") pod \"a2f87461-a7b7-4d97-b76c-b8053d252060\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " Oct 31 20:11:10.575330 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.575230 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca\") pod \"a2f87461-a7b7-4d97-b76c-b8053d252060\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " Oct 31 20:11:10.575330 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.575257 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h\" (UniqueName: \"kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h\") pod \"a2f87461-a7b7-4d97-b76c-b8053d252060\" (UID: \"a2f87461-a7b7-4d97-b76c-b8053d252060\") " Oct 31 20:11:10.575584 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.575557 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca" pod "a2f87461-a7b7-4d97-b76c-b8053d252060" (UID: "a2f87461-a7b7-4d97-b76c-b8053d252060"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:11:10.577465 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.577441 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca" pod "a2f87461-a7b7-4d97-b76c-b8053d252060" (UID: "a2f87461-a7b7-4d97-b76c-b8053d252060"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:11:10.577465 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.577446 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h" pod "a2f87461-a7b7-4d97-b76c-b8053d252060" (UID: "a2f87461-a7b7-4d97-b76c-b8053d252060"). InnerVolumeSpecName "kube-api-access-6d98h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:11:10.676357 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.676272 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.676357 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.676303 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f87461-a7b7-4d97-b76c-b8053d252060-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.676357 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:10.676313 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d98h\" (UniqueName: \"kubernetes.io/projected/a2f87461-a7b7-4d97-b76c-b8053d252060-kube-api-access-6d98h\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:11:11.375018 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:11.374984 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" Oct 31 20:11:11.375234 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:11.374988 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs" event={"ID":"a2f87461-a7b7-4d97-b76c-b8053d252060","Type":"ContainerDied","Data":"642880ec6d2b5c41136f31eced0b6ec23ac428ad618fb8a1af660f623588ceb8"} Oct 31 20:11:11.375234 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:11.375116 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="642880ec6d2b5c41136f31eced0b6ec23ac428ad618fb8a1af660f623588ceb8" Oct 31 20:11:20.440366 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:20.440334 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:11:20.451107 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:20.451084 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:11:24.905006 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:11:24.904967 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:11:52.460397 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.460358 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td"] Oct 31 20:11:52.460950 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.460691 5688 cpu_manager.go:414] "containerMap: removing container" podUID="a2f87461-a7b7-4d97-b76c-b8053d252060" containerName="revoke-keys" Oct 31 20:11:52.460950 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.460701 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f87461-a7b7-4d97-b76c-b8053d252060" containerName="revoke-keys" Oct 31 20:11:52.460950 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.460782 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f87461-a7b7-4d97-b76c-b8053d252060" containerName="revoke-keys" Oct 31 20:11:52.467412 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.465696 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.523551 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.523521 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td"] Oct 31 20:11:52.567324 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.567282 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.567324 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.567329 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.567553 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.567444 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsr2m\" (UniqueName: \"kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.571348 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.571317 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:11:52.571348 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.571329 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:11:52.571508 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.571451 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:11:52.571508 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.571501 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:11:52.667955 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.667916 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.668143 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.668013 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsr2m\" (UniqueName: \"kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.668143 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.668114 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.668724 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.668700 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.670622 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.670597 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.725633 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.725546 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsr2m\" (UniqueName: \"kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m\") pod \"maas-api-revoke-sub-a8d0e5cc-nb5td\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:52.779528 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:52.779475 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:53.151924 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:53.151819 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td"] Oct 31 20:11:53.155169 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:11:53.155141 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56426894_02f1_43e8_8fec_407765c009ab.slice/crio-3d58648d897baa62f94dd0a75ad4a1f9e69b6a394f434c41d8dad60cc6b0a23d WatchSource:0}: Error finding container 3d58648d897baa62f94dd0a75ad4a1f9e69b6a394f434c41d8dad60cc6b0a23d: Status 404 returned error can't find the container with id 3d58648d897baa62f94dd0a75ad4a1f9e69b6a394f434c41d8dad60cc6b0a23d Oct 31 20:11:53.591630 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:53.591586 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" event={"ID":"56426894-02f1-43e8-8fec-407765c009ab","Type":"ContainerStarted","Data":"5f242672682e4dc5336597ba110d0293e0ebaa6212243e4c7ae9c47e3493d559"} Oct 31 20:11:53.592085 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:53.591649 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" event={"ID":"56426894-02f1-43e8-8fec-407765c009ab","Type":"ContainerStarted","Data":"3d58648d897baa62f94dd0a75ad4a1f9e69b6a394f434c41d8dad60cc6b0a23d"} Oct 31 20:11:53.707329 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:53.707117 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" podStartSLOduration=1.707099823 podStartE2EDuration="1.707099823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:53.706664541 +0000 UTC m=+6634.308805683" watchObservedRunningTime="2026-10-31 20:11:53.707099823 +0000 UTC m=+6634.309240966" Oct 31 20:11:54.597823 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:54.597790 5688 generic.go:356] "Generic (PLEG): container finished" podID="56426894-02f1-43e8-8fec-407765c009ab" containerID="5f242672682e4dc5336597ba110d0293e0ebaa6212243e4c7ae9c47e3493d559" exitCode=0 Oct 31 20:11:54.598253 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:54.597876 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" event={"ID":"56426894-02f1-43e8-8fec-407765c009ab","Type":"ContainerDied","Data":"5f242672682e4dc5336597ba110d0293e0ebaa6212243e4c7ae9c47e3493d559"} Oct 31 20:11:54.904719 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:11:54.904639 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:11:55.740461 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.740437 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:55.898286 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.898175 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca\") pod \"56426894-02f1-43e8-8fec-407765c009ab\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " Oct 31 20:11:55.898471 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.898286 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m\" (UniqueName: \"kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m\") pod \"56426894-02f1-43e8-8fec-407765c009ab\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " Oct 31 20:11:55.898471 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.898320 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca\") pod \"56426894-02f1-43e8-8fec-407765c009ab\" (UID: \"56426894-02f1-43e8-8fec-407765c009ab\") " Oct 31 20:11:55.898611 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.898556 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca" pod "56426894-02f1-43e8-8fec-407765c009ab" (UID: "56426894-02f1-43e8-8fec-407765c009ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:11:55.900570 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.900541 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m" pod "56426894-02f1-43e8-8fec-407765c009ab" (UID: "56426894-02f1-43e8-8fec-407765c009ab"). InnerVolumeSpecName "kube-api-access-fsr2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:11:55.900671 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.900623 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca" pod "56426894-02f1-43e8-8fec-407765c009ab" (UID: "56426894-02f1-43e8-8fec-407765c009ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:11:55.999236 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.999168 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56426894-02f1-43e8-8fec-407765c009ab-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:11:55.999236 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.999231 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsr2m\" (UniqueName: \"kubernetes.io/projected/56426894-02f1-43e8-8fec-407765c009ab-kube-api-access-fsr2m\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:11:55.999236 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:55.999244 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56426894-02f1-43e8-8fec-407765c009ab-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:11:56.608240 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:56.608209 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" Oct 31 20:11:56.608419 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:56.608170 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td" event={"ID":"56426894-02f1-43e8-8fec-407765c009ab","Type":"ContainerDied","Data":"3d58648d897baa62f94dd0a75ad4a1f9e69b6a394f434c41d8dad60cc6b0a23d"} Oct 31 20:11:56.608419 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:11:56.608335 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d58648d897baa62f94dd0a75ad4a1f9e69b6a394f434c41d8dad60cc6b0a23d" Oct 31 20:12:03.290664 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.290630 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq"] Oct 31 20:12:03.291156 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.291022 5688 cpu_manager.go:414] "containerMap: removing container" podUID="56426894-02f1-43e8-8fec-407765c009ab" containerName="revoke-keys" Oct 31 20:12:03.291156 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.291034 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56426894-02f1-43e8-8fec-407765c009ab" containerName="revoke-keys" Oct 31 20:12:03.291156 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.291102 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="56426894-02f1-43e8-8fec-407765c009ab" containerName="revoke-keys" Oct 31 20:12:03.294316 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.294299 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.347082 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.347043 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:12:03.396941 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.396869 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:12:03.397139 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.397103 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:12:03.397139 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.397127 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:12:03.398450 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.398433 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq"] Oct 31 20:12:03.461361 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.461328 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.461361 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.461367 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqr9\" (UniqueName: \"kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.461606 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.461488 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.562614 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.562534 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.562614 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.562584 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.562614 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.562603 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqr9\" (UniqueName: \"kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.563260 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.563232 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.565286 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.565259 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.621533 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.621495 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqr9\" (UniqueName: \"kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9\") pod \"maas-api-revoke-sub-44fa7a57-2xrgq\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:03.909629 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:03.909543 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:04.279904 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:04.279870 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq"] Oct 31 20:12:04.283105 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:12:04.283069 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2baaab_af86_420b_89cc_7c7c4195e0eb.slice/crio-7a9f4741a0fc35bcd45a828023c2af183edc2098391e3406f096031c8f81bc68 WatchSource:0}: Error finding container 7a9f4741a0fc35bcd45a828023c2af183edc2098391e3406f096031c8f81bc68: Status 404 returned error can't find the container with id 7a9f4741a0fc35bcd45a828023c2af183edc2098391e3406f096031c8f81bc68 Oct 31 20:12:04.642742 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:04.642708 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" event={"ID":"8a2baaab-af86-420b-89cc-7c7c4195e0eb","Type":"ContainerStarted","Data":"c4a0b43c89d8a6953cc89f1f4f37642600eed8fe00f0596f380cce1334cdbb4a"} Oct 31 20:12:04.643262 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:04.642759 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" event={"ID":"8a2baaab-af86-420b-89cc-7c7c4195e0eb","Type":"ContainerStarted","Data":"7a9f4741a0fc35bcd45a828023c2af183edc2098391e3406f096031c8f81bc68"} Oct 31 20:12:04.758221 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:04.758152 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" podStartSLOduration=1.758139242 podStartE2EDuration="1.758139242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:04.756973343 +0000 UTC m=+6645.359114490" watchObservedRunningTime="2026-10-31 20:12:04.758139242 +0000 UTC m=+6645.360280424" Oct 31 20:12:05.648949 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:05.648910 5688 generic.go:356] "Generic (PLEG): container finished" podID="8a2baaab-af86-420b-89cc-7c7c4195e0eb" containerID="c4a0b43c89d8a6953cc89f1f4f37642600eed8fe00f0596f380cce1334cdbb4a" exitCode=0 Oct 31 20:12:05.649387 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:05.648967 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" event={"ID":"8a2baaab-af86-420b-89cc-7c7c4195e0eb","Type":"ContainerDied","Data":"c4a0b43c89d8a6953cc89f1f4f37642600eed8fe00f0596f380cce1334cdbb4a"} Oct 31 20:12:06.786239 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.786201 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:06.885726 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.885694 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca\") pod \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " Oct 31 20:12:06.885922 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.885737 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9\" (UniqueName: \"kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9\") pod \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " Oct 31 20:12:06.885922 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.885759 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca\") pod \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\" (UID: \"8a2baaab-af86-420b-89cc-7c7c4195e0eb\") " Oct 31 20:12:06.886127 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.886104 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca" pod "8a2baaab-af86-420b-89cc-7c7c4195e0eb" (UID: "8a2baaab-af86-420b-89cc-7c7c4195e0eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:06.887963 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.887940 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9" pod "8a2baaab-af86-420b-89cc-7c7c4195e0eb" (UID: "8a2baaab-af86-420b-89cc-7c7c4195e0eb"). InnerVolumeSpecName "kube-api-access-rjqr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:06.888072 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.887984 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca" pod "8a2baaab-af86-420b-89cc-7c7c4195e0eb" (UID: "8a2baaab-af86-420b-89cc-7c7c4195e0eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:06.986336 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.986305 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:12:06.986336 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.986335 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjqr9\" (UniqueName: \"kubernetes.io/projected/8a2baaab-af86-420b-89cc-7c7c4195e0eb-kube-api-access-rjqr9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:12:06.986547 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:06.986344 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2baaab-af86-420b-89cc-7c7c4195e0eb-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:12:07.658400 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:07.658365 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" event={"ID":"8a2baaab-af86-420b-89cc-7c7c4195e0eb","Type":"ContainerDied","Data":"7a9f4741a0fc35bcd45a828023c2af183edc2098391e3406f096031c8f81bc68"} Oct 31 20:12:07.658545 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:07.658408 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq" Oct 31 20:12:07.658545 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:07.658415 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9f4741a0fc35bcd45a828023c2af183edc2098391e3406f096031c8f81bc68" Oct 31 20:12:24.925272 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:12:24.925233 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:12:31.209269 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:31.208339 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ztb8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:12:31.209269 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:31.208404 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ztb8k" podUID="b1c9fc38-2e60-4b56-a5dd-a223912fc013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:12:51.478749 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.478714 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb"] Oct 31 20:12:51.479229 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.479057 5688 cpu_manager.go:414] "containerMap: removing container" podUID="8a2baaab-af86-420b-89cc-7c7c4195e0eb" containerName="revoke-keys" Oct 31 20:12:51.479229 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.479068 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2baaab-af86-420b-89cc-7c7c4195e0eb" containerName="revoke-keys" Oct 31 20:12:51.479229 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.479167 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2baaab-af86-420b-89cc-7c7c4195e0eb" containerName="revoke-keys" Oct 31 20:12:51.482172 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.482155 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.534792 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.534764 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:12:51.534792 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.534776 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:12:51.534965 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.534856 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:12:51.534965 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.534898 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:12:51.545319 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.545289 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb"] Oct 31 20:12:51.578905 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.578880 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.579024 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.578921 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wlh9\" (UniqueName: \"kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.579067 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.579031 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.679475 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.679447 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wlh9\" (UniqueName: \"kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.679597 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.679533 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.679597 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.679569 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.680113 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.680095 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.681999 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.681979 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.739867 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.739836 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wlh9\" (UniqueName: \"kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9\") pod \"maas-api-revoke-sub-c3d0d789-4jqkb\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:51.794661 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:51.794627 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:52.172030 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:52.171955 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb"] Oct 31 20:12:52.175225 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:12:52.175178 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e58c4f_3312_4f27_957f_a4e46f856954.slice/crio-9ec52c6dfbacba04eb1b02b904761c468b1cc6c495fd00c4f3e0ce2668f5f669 WatchSource:0}: Error finding container 9ec52c6dfbacba04eb1b02b904761c468b1cc6c495fd00c4f3e0ce2668f5f669: Status 404 returned error can't find the container with id 9ec52c6dfbacba04eb1b02b904761c468b1cc6c495fd00c4f3e0ce2668f5f669 Oct 31 20:12:52.913433 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:52.913390 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" event={"ID":"58e58c4f-3312-4f27-957f-a4e46f856954","Type":"ContainerStarted","Data":"a5aafcd2a9278fb17a0b1c512fc2c22675be0d61099d708a526fbe1c08ab610a"} Oct 31 20:12:52.913830 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:52.913456 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" event={"ID":"58e58c4f-3312-4f27-957f-a4e46f856954","Type":"ContainerStarted","Data":"9ec52c6dfbacba04eb1b02b904761c468b1cc6c495fd00c4f3e0ce2668f5f669"} Oct 31 20:12:53.028429 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:53.028368 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" podStartSLOduration=2.028349524 podStartE2EDuration="2.028349524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:53.027648907 +0000 UTC m=+6693.629790051" watchObservedRunningTime="2026-10-31 20:12:53.028349524 +0000 UTC m=+6693.630490669" Oct 31 20:12:53.919270 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:53.919235 5688 generic.go:356] "Generic (PLEG): container finished" podID="58e58c4f-3312-4f27-957f-a4e46f856954" containerID="a5aafcd2a9278fb17a0b1c512fc2c22675be0d61099d708a526fbe1c08ab610a" exitCode=0 Oct 31 20:12:53.919771 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:53.919318 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" event={"ID":"58e58c4f-3312-4f27-957f-a4e46f856954","Type":"ContainerDied","Data":"a5aafcd2a9278fb17a0b1c512fc2c22675be0d61099d708a526fbe1c08ab610a"} Oct 31 20:12:54.904072 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:12:54.904036 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:12:55.052257 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.052232 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:55.108446 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.108417 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca\") pod \"58e58c4f-3312-4f27-957f-a4e46f856954\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " Oct 31 20:12:55.108648 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.108454 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca\") pod \"58e58c4f-3312-4f27-957f-a4e46f856954\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " Oct 31 20:12:55.108648 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.108481 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9\" (UniqueName: \"kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9\") pod \"58e58c4f-3312-4f27-957f-a4e46f856954\" (UID: \"58e58c4f-3312-4f27-957f-a4e46f856954\") " Oct 31 20:12:55.108837 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.108812 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca" pod "58e58c4f-3312-4f27-957f-a4e46f856954" (UID: "58e58c4f-3312-4f27-957f-a4e46f856954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:55.110679 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.110651 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca" pod "58e58c4f-3312-4f27-957f-a4e46f856954" (UID: "58e58c4f-3312-4f27-957f-a4e46f856954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:55.110780 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.110712 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9" pod "58e58c4f-3312-4f27-957f-a4e46f856954" (UID: "58e58c4f-3312-4f27-957f-a4e46f856954"). InnerVolumeSpecName "kube-api-access-7wlh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:55.209773 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.209701 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:12:55.209773 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.209730 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e58c4f-3312-4f27-957f-a4e46f856954-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:12:55.209773 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.209740 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wlh9\" (UniqueName: \"kubernetes.io/projected/58e58c4f-3312-4f27-957f-a4e46f856954-kube-api-access-7wlh9\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:12:55.930098 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.930011 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" event={"ID":"58e58c4f-3312-4f27-957f-a4e46f856954","Type":"ContainerDied","Data":"9ec52c6dfbacba04eb1b02b904761c468b1cc6c495fd00c4f3e0ce2668f5f669"} Oct 31 20:12:55.930098 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.930055 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d0d789-4jqkb" Oct 31 20:12:55.930098 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:12:55.930085 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ec52c6dfbacba04eb1b02b904761c468b1cc6c495fd00c4f3e0ce2668f5f669" Oct 31 20:13:02.274032 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.273998 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng"] Oct 31 20:13:02.274446 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.274359 5688 cpu_manager.go:414] "containerMap: removing container" podUID="58e58c4f-3312-4f27-957f-a4e46f856954" containerName="revoke-keys" Oct 31 20:13:02.274446 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.274371 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e58c4f-3312-4f27-957f-a4e46f856954" containerName="revoke-keys" Oct 31 20:13:02.274446 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.274441 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e58c4f-3312-4f27-957f-a4e46f856954" containerName="revoke-keys" Oct 31 20:13:02.277473 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.277452 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.373140 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.373109 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.373331 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.373149 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf678\" (UniqueName: \"kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.373331 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.373296 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.379476 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.379456 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:13:02.379534 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.379469 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:13:02.379534 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.379520 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:13:02.379621 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.379550 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:13:02.380944 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.380925 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng"] Oct 31 20:13:02.474807 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.474775 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf678\" (UniqueName: \"kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.474964 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.474843 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.474964 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.474877 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.475530 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.475509 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.477371 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.477348 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.531866 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.531805 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf678\" (UniqueName: \"kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678\") pod \"maas-api-revoke-sub-d65e62dd-q8mng\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.590023 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.589982 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:02.969027 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:02.968954 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng"] Oct 31 20:13:02.972464 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:13:02.972437 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30ad12a_d428_4567_9f73_002e332c9dbd.slice/crio-48f525e91e3a84de0bc55234c89b92d40851c2dd681a00bec23f9dd5853930fd WatchSource:0}: Error finding container 48f525e91e3a84de0bc55234c89b92d40851c2dd681a00bec23f9dd5853930fd: Status 404 returned error can't find the container with id 48f525e91e3a84de0bc55234c89b92d40851c2dd681a00bec23f9dd5853930fd Oct 31 20:13:03.966569 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:03.966538 5688 generic.go:356] "Generic (PLEG): container finished" podID="c30ad12a-d428-4567-9f73-002e332c9dbd" containerID="c257425272e002caf6b461f1d37f0b2f866af5a58d0f13dd82ba4fad95d3b714" exitCode=0 Oct 31 20:13:03.966968 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:03.966584 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" event={"ID":"c30ad12a-d428-4567-9f73-002e332c9dbd","Type":"ContainerDied","Data":"c257425272e002caf6b461f1d37f0b2f866af5a58d0f13dd82ba4fad95d3b714"} Oct 31 20:13:03.966968 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:03.966622 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" event={"ID":"c30ad12a-d428-4567-9f73-002e332c9dbd","Type":"ContainerStarted","Data":"48f525e91e3a84de0bc55234c89b92d40851c2dd681a00bec23f9dd5853930fd"} Oct 31 20:13:05.142160 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.142131 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:05.202805 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.202770 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678\" (UniqueName: \"kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678\") pod \"c30ad12a-d428-4567-9f73-002e332c9dbd\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " Oct 31 20:13:05.203013 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.202851 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca\") pod \"c30ad12a-d428-4567-9f73-002e332c9dbd\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " Oct 31 20:13:05.203013 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.202958 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca\") pod \"c30ad12a-d428-4567-9f73-002e332c9dbd\" (UID: \"c30ad12a-d428-4567-9f73-002e332c9dbd\") " Oct 31 20:13:05.203587 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.203554 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca" pod "c30ad12a-d428-4567-9f73-002e332c9dbd" (UID: "c30ad12a-d428-4567-9f73-002e332c9dbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:05.205141 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.205116 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678" pod "c30ad12a-d428-4567-9f73-002e332c9dbd" (UID: "c30ad12a-d428-4567-9f73-002e332c9dbd"). InnerVolumeSpecName "kube-api-access-gf678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:05.205666 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.205643 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca" pod "c30ad12a-d428-4567-9f73-002e332c9dbd" (UID: "c30ad12a-d428-4567-9f73-002e332c9dbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:05.304241 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.304176 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:13:05.304241 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.304235 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf678\" (UniqueName: \"kubernetes.io/projected/c30ad12a-d428-4567-9f73-002e332c9dbd-kube-api-access-gf678\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:13:05.304241 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.304250 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30ad12a-d428-4567-9f73-002e332c9dbd-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:13:05.999533 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.999490 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" Oct 31 20:13:05.999533 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.999517 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65e62dd-q8mng" event={"ID":"c30ad12a-d428-4567-9f73-002e332c9dbd","Type":"ContainerDied","Data":"48f525e91e3a84de0bc55234c89b92d40851c2dd681a00bec23f9dd5853930fd"} Oct 31 20:13:05.999836 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:05.999572 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f525e91e3a84de0bc55234c89b92d40851c2dd681a00bec23f9dd5853930fd" Oct 31 20:13:24.904769 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:13:24.904725 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:13:47.154369 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:47.154322 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-85vdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:13:47.154940 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:47.154424 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-85vdx" podUID="3affcc65-df38-4880-8581-75f2dffc6b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:13:47.196211 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:47.195268 5688 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp75p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:13:47.196211 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:47.195355 5688 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp75p" podUID="ea3982da-e2ec-4ffb-9b3f-a805c03bcbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:13:53.721791 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.721756 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc"] Oct 31 20:13:53.722239 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.722152 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c30ad12a-d428-4567-9f73-002e332c9dbd" containerName="revoke-keys" Oct 31 20:13:53.722239 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.722165 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30ad12a-d428-4567-9f73-002e332c9dbd" containerName="revoke-keys" Oct 31 20:13:53.722338 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.722247 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30ad12a-d428-4567-9f73-002e332c9dbd" containerName="revoke-keys" Oct 31 20:13:53.725285 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.725268 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.741993 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.741963 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.742109 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.742019 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kf2g\" (UniqueName: \"kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.742109 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.742092 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.778132 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.778106 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:13:53.827769 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.827726 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:13:53.827769 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.827736 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:13:53.827987 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.827778 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:13:53.829243 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.829210 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc"] Oct 31 20:13:53.842706 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.842677 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.842815 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.842738 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kf2g\" (UniqueName: \"kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.842815 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.842760 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.843423 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.843406 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.845147 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.845129 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:53.900444 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:53.900411 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kf2g\" (UniqueName: \"kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g\") pod \"maas-api-revoke-sub-f18900b6-wgdxc\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:54.037934 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:54.037893 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:54.409465 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:54.409394 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc"] Oct 31 20:13:54.412447 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:13:54.412423 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77b78d6_f806_4700_a349_4427a0fb1025.slice/crio-4bd88820e962a23a7f4d52d29407aef991d14d114434335fc52da6b1708b58ed WatchSource:0}: Error finding container 4bd88820e962a23a7f4d52d29407aef991d14d114434335fc52da6b1708b58ed: Status 404 returned error can't find the container with id 4bd88820e962a23a7f4d52d29407aef991d14d114434335fc52da6b1708b58ed Oct 31 20:13:54.903171 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:13:54.903143 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:13:55.297567 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:55.297537 5688 generic.go:356] "Generic (PLEG): container finished" podID="c77b78d6-f806-4700-a349-4427a0fb1025" containerID="32ab9f6915167e3ee71093bf2923f7390fbb7da496d2f46b55bca4e4c2997748" exitCode=0 Oct 31 20:13:55.297777 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:55.297593 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" event={"ID":"c77b78d6-f806-4700-a349-4427a0fb1025","Type":"ContainerDied","Data":"32ab9f6915167e3ee71093bf2923f7390fbb7da496d2f46b55bca4e4c2997748"} Oct 31 20:13:55.297777 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:55.297642 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" event={"ID":"c77b78d6-f806-4700-a349-4427a0fb1025","Type":"ContainerStarted","Data":"4bd88820e962a23a7f4d52d29407aef991d14d114434335fc52da6b1708b58ed"} Oct 31 20:13:56.428986 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.428963 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:56.459529 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.459507 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca\") pod \"c77b78d6-f806-4700-a349-4427a0fb1025\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " Oct 31 20:13:56.459643 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.459558 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g\" (UniqueName: \"kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g\") pod \"c77b78d6-f806-4700-a349-4427a0fb1025\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " Oct 31 20:13:56.459829 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.459809 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca" pod "c77b78d6-f806-4700-a349-4427a0fb1025" (UID: "c77b78d6-f806-4700-a349-4427a0fb1025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:56.461673 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.461645 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g" pod "c77b78d6-f806-4700-a349-4427a0fb1025" (UID: "c77b78d6-f806-4700-a349-4427a0fb1025"). InnerVolumeSpecName "kube-api-access-2kf2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:56.560251 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.560165 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca\") pod \"c77b78d6-f806-4700-a349-4427a0fb1025\" (UID: \"c77b78d6-f806-4700-a349-4427a0fb1025\") " Oct 31 20:13:56.560559 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.560523 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kf2g\" (UniqueName: \"kubernetes.io/projected/c77b78d6-f806-4700-a349-4427a0fb1025-kube-api-access-2kf2g\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:13:56.560692 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.560565 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:13:56.562442 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.562411 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca" pod "c77b78d6-f806-4700-a349-4427a0fb1025" (UID: "c77b78d6-f806-4700-a349-4427a0fb1025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:56.660939 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:56.660912 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77b78d6-f806-4700-a349-4427a0fb1025-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:13:57.307956 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:57.307929 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" Oct 31 20:13:57.308140 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:57.307928 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18900b6-wgdxc" event={"ID":"c77b78d6-f806-4700-a349-4427a0fb1025","Type":"ContainerDied","Data":"4bd88820e962a23a7f4d52d29407aef991d14d114434335fc52da6b1708b58ed"} Oct 31 20:13:57.308140 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:13:57.308064 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd88820e962a23a7f4d52d29407aef991d14d114434335fc52da6b1708b58ed" Oct 31 20:14:04.776979 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.776904 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng"] Oct 31 20:14:04.777372 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.777251 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c77b78d6-f806-4700-a349-4427a0fb1025" containerName="revoke-keys" Oct 31 20:14:04.777372 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.777262 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77b78d6-f806-4700-a349-4427a0fb1025" containerName="revoke-keys" Oct 31 20:14:04.777372 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.777324 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77b78d6-f806-4700-a349-4427a0fb1025" containerName="revoke-keys" Oct 31 20:14:04.780393 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.780377 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.825266 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.825233 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqf8l\" (UniqueName: \"kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.825410 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.825293 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.825461 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.825410 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.832815 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.832794 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:14:04.882467 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.882442 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:04.882593 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.882470 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:14:04.882634 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.882601 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:14:04.884349 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.884332 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng"] Oct 31 20:14:04.926235 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.926213 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqf8l\" (UniqueName: \"kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.926341 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.926257 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.926341 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.926276 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.926952 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.926932 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.928806 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.928784 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:04.982747 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:04.982722 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqf8l\" (UniqueName: \"kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l\") pod \"maas-api-revoke-sub-9ce7568e-nmlng\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:05.092735 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:05.092665 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:05.461665 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:05.461640 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng"] Oct 31 20:14:05.464311 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:14:05.464282 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a574ff_526d_4931_9d9f_07fdfb13f9ec.slice/crio-27e0b571ace4866953db35cde082148d0c169075321d6b55db724bec2908ee31 WatchSource:0}: Error finding container 27e0b571ace4866953db35cde082148d0c169075321d6b55db724bec2908ee31: Status 404 returned error can't find the container with id 27e0b571ace4866953db35cde082148d0c169075321d6b55db724bec2908ee31 Oct 31 20:14:06.351082 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:06.351052 5688 generic.go:356] "Generic (PLEG): container finished" podID="e9a574ff-526d-4931-9d9f-07fdfb13f9ec" containerID="984f01a24324414ed544aad55f90189c552ca933b96bcb576b47012c851537c7" exitCode=0 Oct 31 20:14:06.351466 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:06.351090 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" event={"ID":"e9a574ff-526d-4931-9d9f-07fdfb13f9ec","Type":"ContainerDied","Data":"984f01a24324414ed544aad55f90189c552ca933b96bcb576b47012c851537c7"} Oct 31 20:14:06.351466 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:06.351158 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" event={"ID":"e9a574ff-526d-4931-9d9f-07fdfb13f9ec","Type":"ContainerStarted","Data":"27e0b571ace4866953db35cde082148d0c169075321d6b55db724bec2908ee31"} Oct 31 20:14:07.482759 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.482735 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:07.545477 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.545452 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca\") pod \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " Oct 31 20:14:07.545597 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.545492 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l\" (UniqueName: \"kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l\") pod \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " Oct 31 20:14:07.545597 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.545525 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca\") pod \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\" (UID: \"e9a574ff-526d-4931-9d9f-07fdfb13f9ec\") " Oct 31 20:14:07.545767 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.545747 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca" pod "e9a574ff-526d-4931-9d9f-07fdfb13f9ec" (UID: "e9a574ff-526d-4931-9d9f-07fdfb13f9ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:07.547529 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.547501 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca" pod "e9a574ff-526d-4931-9d9f-07fdfb13f9ec" (UID: "e9a574ff-526d-4931-9d9f-07fdfb13f9ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:07.547618 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.547583 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l" pod "e9a574ff-526d-4931-9d9f-07fdfb13f9ec" (UID: "e9a574ff-526d-4931-9d9f-07fdfb13f9ec"). InnerVolumeSpecName "kube-api-access-tqf8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:07.646535 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.646466 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:07.646535 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.646495 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqf8l\" (UniqueName: \"kubernetes.io/projected/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-kube-api-access-tqf8l\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:07.646535 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:07.646509 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a574ff-526d-4931-9d9f-07fdfb13f9ec-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:08.361475 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:08.361449 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" Oct 31 20:14:08.361648 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:08.361482 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce7568e-nmlng" event={"ID":"e9a574ff-526d-4931-9d9f-07fdfb13f9ec","Type":"ContainerDied","Data":"27e0b571ace4866953db35cde082148d0c169075321d6b55db724bec2908ee31"} Oct 31 20:14:08.361648 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:08.361547 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e0b571ace4866953db35cde082148d0c169075321d6b55db724bec2908ee31" Oct 31 20:14:15.900906 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.900873 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r"] Oct 31 20:14:15.901300 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.901205 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e9a574ff-526d-4931-9d9f-07fdfb13f9ec" containerName="revoke-keys" Oct 31 20:14:15.901300 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.901216 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a574ff-526d-4931-9d9f-07fdfb13f9ec" containerName="revoke-keys" Oct 31 20:14:15.901300 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.901298 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a574ff-526d-4931-9d9f-07fdfb13f9ec" containerName="revoke-keys" Oct 31 20:14:15.904174 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.904151 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:15.946694 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.946658 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:15.946694 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.946698 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smj5q\" (UniqueName: \"kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:15.946694 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.946718 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:15.957411 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.957387 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:14:15.957522 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:15.957470 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:14:16.007178 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.007152 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:14:16.007321 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.007205 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:16.008831 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.008813 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r"] Oct 31 20:14:16.047471 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.047447 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.047601 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.047481 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smj5q\" (UniqueName: \"kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.047601 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.047501 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.048134 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.048116 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.049836 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.049821 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.103906 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.103887 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smj5q\" (UniqueName: \"kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q\") pod \"maas-api-revoke-sub-0e6d39b1-knm7r\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.216148 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.216084 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:16.583676 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:16.583646 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r"] Oct 31 20:14:16.586730 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:14:16.586700 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5f96c4_67a5_493a_9c10_5966f2af8d61.slice/crio-61161ae8003efb85734e4f37ac4896df9d96070a76d0ee51cc6f98ce89fb9614 WatchSource:0}: Error finding container 61161ae8003efb85734e4f37ac4896df9d96070a76d0ee51cc6f98ce89fb9614: Status 404 returned error can't find the container with id 61161ae8003efb85734e4f37ac4896df9d96070a76d0ee51cc6f98ce89fb9614 Oct 31 20:14:17.423554 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:17.423524 5688 generic.go:356] "Generic (PLEG): container finished" podID="3b5f96c4-67a5-493a-9c10-5966f2af8d61" containerID="a99a86d8a8ee4a9054c7a9f12add1909bb0c4211402cad608b665308bb10701d" exitCode=0 Oct 31 20:14:17.423931 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:17.423606 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" event={"ID":"3b5f96c4-67a5-493a-9c10-5966f2af8d61","Type":"ContainerDied","Data":"a99a86d8a8ee4a9054c7a9f12add1909bb0c4211402cad608b665308bb10701d"} Oct 31 20:14:17.423931 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:17.423646 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" event={"ID":"3b5f96c4-67a5-493a-9c10-5966f2af8d61","Type":"ContainerStarted","Data":"61161ae8003efb85734e4f37ac4896df9d96070a76d0ee51cc6f98ce89fb9614"} Oct 31 20:14:18.554335 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.554308 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:18.666197 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.666161 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q\" (UniqueName: \"kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q\") pod \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " Oct 31 20:14:18.666321 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.666220 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca\") pod \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " Oct 31 20:14:18.666321 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.666270 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca\") pod \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\" (UID: \"3b5f96c4-67a5-493a-9c10-5966f2af8d61\") " Oct 31 20:14:18.666550 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.666533 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca" pod "3b5f96c4-67a5-493a-9c10-5966f2af8d61" (UID: "3b5f96c4-67a5-493a-9c10-5966f2af8d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:18.668746 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.668544 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca" pod "3b5f96c4-67a5-493a-9c10-5966f2af8d61" (UID: "3b5f96c4-67a5-493a-9c10-5966f2af8d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:18.668746 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.668699 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q" pod "3b5f96c4-67a5-493a-9c10-5966f2af8d61" (UID: "3b5f96c4-67a5-493a-9c10-5966f2af8d61"). InnerVolumeSpecName "kube-api-access-smj5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:18.767282 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.767255 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smj5q\" (UniqueName: \"kubernetes.io/projected/3b5f96c4-67a5-493a-9c10-5966f2af8d61-kube-api-access-smj5q\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:18.767282 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.767280 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:18.767443 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:18.767290 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5f96c4-67a5-493a-9c10-5966f2af8d61-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:19.433387 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:19.433353 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" Oct 31 20:14:19.433561 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:19.433379 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e6d39b1-knm7r" event={"ID":"3b5f96c4-67a5-493a-9c10-5966f2af8d61","Type":"ContainerDied","Data":"61161ae8003efb85734e4f37ac4896df9d96070a76d0ee51cc6f98ce89fb9614"} Oct 31 20:14:19.433561 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:19.433431 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61161ae8003efb85734e4f37ac4896df9d96070a76d0ee51cc6f98ce89fb9614" Oct 31 20:14:24.903605 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:14:24.903568 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:14:26.822141 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.822110 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv"] Oct 31 20:14:26.822690 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.822671 5688 cpu_manager.go:414] "containerMap: removing container" podUID="3b5f96c4-67a5-493a-9c10-5966f2af8d61" containerName="revoke-keys" Oct 31 20:14:26.822736 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.822693 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5f96c4-67a5-493a-9c10-5966f2af8d61" containerName="revoke-keys" Oct 31 20:14:26.822784 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.822774 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5f96c4-67a5-493a-9c10-5966f2af8d61" containerName="revoke-keys" Oct 31 20:14:26.826280 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.826264 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:26.875211 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.875162 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv"] Oct 31 20:14:26.923380 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.923357 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:26.923514 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.923408 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:14:26.923514 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.923423 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:14:26.927770 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.927758 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:14:26.933100 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.933083 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:26.933162 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.933146 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzm5\" (UniqueName: \"kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:26.933264 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:26.933250 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.034103 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.034076 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzm5\" (UniqueName: \"kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.034271 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.034114 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.034271 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.034154 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.034777 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.034749 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.036596 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.036578 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.092496 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.092452 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzm5\" (UniqueName: \"kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5\") pod \"maas-api-revoke-sub-8292ee71-648tv\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.138106 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.138086 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:27.513092 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:27.513064 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv"] Oct 31 20:14:27.516512 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:14:27.516486 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6422ed4b_0593_4817_bde3_d7fb7f648bb2.slice/crio-a550710362b3209bfee86f7696b7d7b8454957bf96a7459534bfde8eb9da3ece WatchSource:0}: Error finding container a550710362b3209bfee86f7696b7d7b8454957bf96a7459534bfde8eb9da3ece: Status 404 returned error can't find the container with id a550710362b3209bfee86f7696b7d7b8454957bf96a7459534bfde8eb9da3ece Oct 31 20:14:28.475460 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:28.475427 5688 generic.go:356] "Generic (PLEG): container finished" podID="6422ed4b-0593-4817-bde3-d7fb7f648bb2" containerID="40c01444660081dea42cfe0c45b0f165f065fa3b3ebcc880e8d285998a1f5ce4" exitCode=0 Oct 31 20:14:28.475818 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:28.475511 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" event={"ID":"6422ed4b-0593-4817-bde3-d7fb7f648bb2","Type":"ContainerDied","Data":"40c01444660081dea42cfe0c45b0f165f065fa3b3ebcc880e8d285998a1f5ce4"} Oct 31 20:14:28.475818 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:28.475564 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" event={"ID":"6422ed4b-0593-4817-bde3-d7fb7f648bb2","Type":"ContainerStarted","Data":"a550710362b3209bfee86f7696b7d7b8454957bf96a7459534bfde8eb9da3ece"} Oct 31 20:14:29.614838 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.614816 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:29.656195 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.656168 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca\") pod \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " Oct 31 20:14:29.656318 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.656240 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca\") pod \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " Oct 31 20:14:29.656318 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.656301 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5\" (UniqueName: \"kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5\") pod \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\" (UID: \"6422ed4b-0593-4817-bde3-d7fb7f648bb2\") " Oct 31 20:14:29.656627 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.656597 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca" pod "6422ed4b-0593-4817-bde3-d7fb7f648bb2" (UID: "6422ed4b-0593-4817-bde3-d7fb7f648bb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:29.658330 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.658310 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca" pod "6422ed4b-0593-4817-bde3-d7fb7f648bb2" (UID: "6422ed4b-0593-4817-bde3-d7fb7f648bb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:29.658447 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.658435 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5" pod "6422ed4b-0593-4817-bde3-d7fb7f648bb2" (UID: "6422ed4b-0593-4817-bde3-d7fb7f648bb2"). InnerVolumeSpecName "kube-api-access-2dzm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:29.756961 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.756939 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzm5\" (UniqueName: \"kubernetes.io/projected/6422ed4b-0593-4817-bde3-d7fb7f648bb2-kube-api-access-2dzm5\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:29.756961 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.756962 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:29.757112 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:29.756971 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ed4b-0593-4817-bde3-d7fb7f648bb2-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:30.486559 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:30.486526 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" event={"ID":"6422ed4b-0593-4817-bde3-d7fb7f648bb2","Type":"ContainerDied","Data":"a550710362b3209bfee86f7696b7d7b8454957bf96a7459534bfde8eb9da3ece"} Oct 31 20:14:30.486742 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:30.486566 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8292ee71-648tv" Oct 31 20:14:30.486742 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:30.486570 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a550710362b3209bfee86f7696b7d7b8454957bf96a7459534bfde8eb9da3ece" Oct 31 20:14:37.595981 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.595951 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4"] Oct 31 20:14:37.596429 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.596293 5688 cpu_manager.go:414] "containerMap: removing container" podUID="6422ed4b-0593-4817-bde3-d7fb7f648bb2" containerName="revoke-keys" Oct 31 20:14:37.596429 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.596305 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6422ed4b-0593-4817-bde3-d7fb7f648bb2" containerName="revoke-keys" Oct 31 20:14:37.596429 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.596387 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="6422ed4b-0593-4817-bde3-d7fb7f648bb2" containerName="revoke-keys" Oct 31 20:14:37.599532 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.599518 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.640216 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.640168 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.640352 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.640225 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.640352 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.640242 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvr96\" (UniqueName: \"kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.703678 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.703650 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:14:37.703678 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.703669 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:37.703877 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.703755 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:14:37.703877 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.703761 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:14:37.705693 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.705675 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4"] Oct 31 20:14:37.741456 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.741428 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.741610 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.741461 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvr96\" (UniqueName: \"kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.741610 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.741574 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.742136 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.742113 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.743730 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.743711 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.799545 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.799516 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvr96\" (UniqueName: \"kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96\") pod \"maas-api-revoke-sub-058ab0b2-dlln4\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:37.912050 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:37.911999 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:38.280672 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:38.280644 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4"] Oct 31 20:14:38.283848 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:14:38.283818 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4e8554_7182_4fe6_b503_c78653b319d2.slice/crio-5d958572e84ca173214e526349f9174af7b32f63c73dcbc91837aed2aadba46c WatchSource:0}: Error finding container 5d958572e84ca173214e526349f9174af7b32f63c73dcbc91837aed2aadba46c: Status 404 returned error can't find the container with id 5d958572e84ca173214e526349f9174af7b32f63c73dcbc91837aed2aadba46c Oct 31 20:14:38.528871 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:38.528837 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" event={"ID":"4f4e8554-7182-4fe6-b503-c78653b319d2","Type":"ContainerStarted","Data":"6a4eaf930bcc58efc2adca92b97b44a0487376ec95d6e3e9494dc2161f5198d9"} Oct 31 20:14:38.528985 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:38.528891 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" event={"ID":"4f4e8554-7182-4fe6-b503-c78653b319d2","Type":"ContainerStarted","Data":"5d958572e84ca173214e526349f9174af7b32f63c73dcbc91837aed2aadba46c"} Oct 31 20:14:38.644254 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:38.644063 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" podStartSLOduration=1.644048975 podStartE2EDuration="1.644048975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:14:38.643251305 +0000 UTC m=+6799.245392449" watchObservedRunningTime="2026-10-31 20:14:38.644048975 +0000 UTC m=+6799.246190120" Oct 31 20:14:39.534650 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:39.534616 5688 generic.go:356] "Generic (PLEG): container finished" podID="4f4e8554-7182-4fe6-b503-c78653b319d2" containerID="6a4eaf930bcc58efc2adca92b97b44a0487376ec95d6e3e9494dc2161f5198d9" exitCode=0 Oct 31 20:14:39.534810 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:39.534705 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" event={"ID":"4f4e8554-7182-4fe6-b503-c78653b319d2","Type":"ContainerDied","Data":"6a4eaf930bcc58efc2adca92b97b44a0487376ec95d6e3e9494dc2161f5198d9"} Oct 31 20:14:40.668526 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.668502 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:40.765958 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.765928 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca\") pod \"4f4e8554-7182-4fe6-b503-c78653b319d2\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " Oct 31 20:14:40.766097 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.765973 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96\" (UniqueName: \"kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96\") pod \"4f4e8554-7182-4fe6-b503-c78653b319d2\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " Oct 31 20:14:40.766097 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.765999 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca\") pod \"4f4e8554-7182-4fe6-b503-c78653b319d2\" (UID: \"4f4e8554-7182-4fe6-b503-c78653b319d2\") " Oct 31 20:14:40.766380 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.766360 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca" pod "4f4e8554-7182-4fe6-b503-c78653b319d2" (UID: "4f4e8554-7182-4fe6-b503-c78653b319d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:40.768178 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.768149 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca" pod "4f4e8554-7182-4fe6-b503-c78653b319d2" (UID: "4f4e8554-7182-4fe6-b503-c78653b319d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:40.768327 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.768309 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96" pod "4f4e8554-7182-4fe6-b503-c78653b319d2" (UID: "4f4e8554-7182-4fe6-b503-c78653b319d2"). InnerVolumeSpecName "kube-api-access-tvr96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:40.867203 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.867112 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvr96\" (UniqueName: \"kubernetes.io/projected/4f4e8554-7182-4fe6-b503-c78653b319d2-kube-api-access-tvr96\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:40.867203 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.867137 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:40.867203 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:40.867151 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4e8554-7182-4fe6-b503-c78653b319d2-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:41.544977 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:41.544947 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" Oct 31 20:14:41.545162 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:41.544994 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058ab0b2-dlln4" event={"ID":"4f4e8554-7182-4fe6-b503-c78653b319d2","Type":"ContainerDied","Data":"5d958572e84ca173214e526349f9174af7b32f63c73dcbc91837aed2aadba46c"} Oct 31 20:14:41.545162 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:41.545045 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d958572e84ca173214e526349f9174af7b32f63c73dcbc91837aed2aadba46c" Oct 31 20:14:48.597274 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.597242 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj"] Oct 31 20:14:48.597692 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.597569 5688 cpu_manager.go:414] "containerMap: removing container" podUID="4f4e8554-7182-4fe6-b503-c78653b319d2" containerName="revoke-keys" Oct 31 20:14:48.597692 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.597583 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4e8554-7182-4fe6-b503-c78653b319d2" containerName="revoke-keys" Oct 31 20:14:48.597692 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.597646 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4e8554-7182-4fe6-b503-c78653b319d2" containerName="revoke-keys" Oct 31 20:14:48.600534 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.600518 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.652985 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.652962 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:14:48.702779 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.702750 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:48.702898 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.702781 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:14:48.702941 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.702907 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:14:48.704866 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.704848 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj"] Oct 31 20:14:48.729375 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.729349 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.729484 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.729378 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqnx\" (UniqueName: \"kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.729484 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.729420 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.829935 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.829911 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.830050 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.829975 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.830050 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.829995 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqnx\" (UniqueName: \"kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.830514 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.830494 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.832398 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.832379 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.886684 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.886636 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqnx\" (UniqueName: \"kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx\") pod \"maas-api-revoke-sub-5427ae91-86vmj\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:48.912748 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:48.912732 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:49.280775 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:49.280739 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj"] Oct 31 20:14:49.283666 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:14:49.283634 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df9228b_5cbb_4c79_9070_8663cb71098a.slice/crio-7c4c63c1b9af35384ea62f2451aca0bea487ab69397a78226a4df4ac539d5ab4 WatchSource:0}: Error finding container 7c4c63c1b9af35384ea62f2451aca0bea487ab69397a78226a4df4ac539d5ab4: Status 404 returned error can't find the container with id 7c4c63c1b9af35384ea62f2451aca0bea487ab69397a78226a4df4ac539d5ab4 Oct 31 20:14:49.583161 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:49.583129 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" event={"ID":"9df9228b-5cbb-4c79-9070-8663cb71098a","Type":"ContainerStarted","Data":"efddc40eb299b647103d9cb54effa08217ae4eba7d7d6c2ebb05700d0fc54e1a"} Oct 31 20:14:49.583310 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:49.583200 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" event={"ID":"9df9228b-5cbb-4c79-9070-8663cb71098a","Type":"ContainerStarted","Data":"7c4c63c1b9af35384ea62f2451aca0bea487ab69397a78226a4df4ac539d5ab4"} Oct 31 20:14:49.702453 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:49.702405 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" podStartSLOduration=1.7023918789999999 podStartE2EDuration="1.702391879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:14:49.700044514 +0000 UTC m=+6810.302185658" watchObservedRunningTime="2026-10-31 20:14:49.702391879 +0000 UTC m=+6810.304533022" Oct 31 20:14:50.597846 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:50.597807 5688 generic.go:356] "Generic (PLEG): container finished" podID="9df9228b-5cbb-4c79-9070-8663cb71098a" containerID="efddc40eb299b647103d9cb54effa08217ae4eba7d7d6c2ebb05700d0fc54e1a" exitCode=0 Oct 31 20:14:50.598025 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:50.597896 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" event={"ID":"9df9228b-5cbb-4c79-9070-8663cb71098a","Type":"ContainerDied","Data":"efddc40eb299b647103d9cb54effa08217ae4eba7d7d6c2ebb05700d0fc54e1a"} Oct 31 20:14:51.913301 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:51.913257 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:51.983208 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:51.982451 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca\") pod \"9df9228b-5cbb-4c79-9070-8663cb71098a\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " Oct 31 20:14:51.983208 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:51.982520 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx\" (UniqueName: \"kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx\") pod \"9df9228b-5cbb-4c79-9070-8663cb71098a\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " Oct 31 20:14:51.983208 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:51.982608 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca\") pod \"9df9228b-5cbb-4c79-9070-8663cb71098a\" (UID: \"9df9228b-5cbb-4c79-9070-8663cb71098a\") " Oct 31 20:14:51.990220 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:51.983926 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca" pod "9df9228b-5cbb-4c79-9070-8663cb71098a" (UID: "9df9228b-5cbb-4c79-9070-8663cb71098a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:51.999352 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:51.998307 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca" pod "9df9228b-5cbb-4c79-9070-8663cb71098a" (UID: "9df9228b-5cbb-4c79-9070-8663cb71098a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:52.006204 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.002883 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx" pod "9df9228b-5cbb-4c79-9070-8663cb71098a" (UID: "9df9228b-5cbb-4c79-9070-8663cb71098a"). InnerVolumeSpecName "kube-api-access-wtqnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:52.085728 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.085690 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:52.085992 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.085971 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df9228b-5cbb-4c79-9070-8663cb71098a-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:52.106277 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.106246 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtqnx\" (UniqueName: \"kubernetes.io/projected/9df9228b-5cbb-4c79-9070-8663cb71098a-kube-api-access-wtqnx\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:14:52.613821 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.613779 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" event={"ID":"9df9228b-5cbb-4c79-9070-8663cb71098a","Type":"ContainerDied","Data":"7c4c63c1b9af35384ea62f2451aca0bea487ab69397a78226a4df4ac539d5ab4"} Oct 31 20:14:52.613993 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.613847 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4c63c1b9af35384ea62f2451aca0bea487ab69397a78226a4df4ac539d5ab4" Oct 31 20:14:52.613993 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:52.613860 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5427ae91-86vmj" Oct 31 20:14:54.903737 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:14:54.903696 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:14:59.675220 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.675177 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5"] Oct 31 20:14:59.675693 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.675644 5688 cpu_manager.go:414] "containerMap: removing container" podUID="9df9228b-5cbb-4c79-9070-8663cb71098a" containerName="revoke-keys" Oct 31 20:14:59.675693 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.675664 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df9228b-5cbb-4c79-9070-8663cb71098a" containerName="revoke-keys" Oct 31 20:14:59.675782 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.675759 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df9228b-5cbb-4c79-9070-8663cb71098a" containerName="revoke-keys" Oct 31 20:14:59.678728 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.678707 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.734465 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.734091 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:59.734465 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.734317 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:14:59.734465 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.734389 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:14:59.737094 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.737069 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5"] Oct 31 20:14:59.765694 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.765667 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.765810 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.765695 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.765810 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.765719 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwpb\" (UniqueName: \"kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.784198 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.784166 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:14:59.866861 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.866835 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.866979 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.866869 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.866979 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.866901 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwpb\" (UniqueName: \"kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.867491 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.867472 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.869323 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.869308 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.924120 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.924095 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwpb\" (UniqueName: \"kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb\") pod \"maas-api-revoke-sub-6512a862-hrfm5\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:14:59.991309 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:14:59.991292 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:15:00.361240 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:00.361216 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5"] Oct 31 20:15:00.363353 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:15:00.363324 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7624aaf_8c64_4315_b1e2_635697aa8820.slice/crio-881347cdb661ebf5c77008263d586a7a27c8a6daca2805d79b91fe71c495f0c2 WatchSource:0}: Error finding container 881347cdb661ebf5c77008263d586a7a27c8a6daca2805d79b91fe71c495f0c2: Status 404 returned error can't find the container with id 881347cdb661ebf5c77008263d586a7a27c8a6daca2805d79b91fe71c495f0c2 Oct 31 20:15:00.651060 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:00.651003 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" event={"ID":"d7624aaf-8c64-4315-b1e2-635697aa8820","Type":"ContainerStarted","Data":"45d7f8a51bbcc01c460e2b2a086cd2c51d64e7dd82ff4474cad26dd545ee8bb5"} Oct 31 20:15:00.651060 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:00.651049 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" event={"ID":"d7624aaf-8c64-4315-b1e2-635697aa8820","Type":"ContainerStarted","Data":"881347cdb661ebf5c77008263d586a7a27c8a6daca2805d79b91fe71c495f0c2"} Oct 31 20:15:00.767167 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:00.767124 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" podStartSLOduration=1.767111726 podStartE2EDuration="1.767111726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:00.764862989 +0000 UTC m=+6821.367004132" watchObservedRunningTime="2026-10-31 20:15:00.767111726 +0000 UTC m=+6821.369252874" Oct 31 20:15:01.658581 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:01.658492 5688 generic.go:356] "Generic (PLEG): container finished" podID="d7624aaf-8c64-4315-b1e2-635697aa8820" containerID="45d7f8a51bbcc01c460e2b2a086cd2c51d64e7dd82ff4474cad26dd545ee8bb5" exitCode=0 Oct 31 20:15:01.658581 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:01.658531 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" event={"ID":"d7624aaf-8c64-4315-b1e2-635697aa8820","Type":"ContainerDied","Data":"45d7f8a51bbcc01c460e2b2a086cd2c51d64e7dd82ff4474cad26dd545ee8bb5"} Oct 31 20:15:02.930473 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:02.930451 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:15:03.008362 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.007836 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca\") pod \"d7624aaf-8c64-4315-b1e2-635697aa8820\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " Oct 31 20:15:03.008621 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.008557 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb\" (UniqueName: \"kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb\") pod \"d7624aaf-8c64-4315-b1e2-635697aa8820\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " Oct 31 20:15:03.008821 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.008773 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca\") pod \"d7624aaf-8c64-4315-b1e2-635697aa8820\" (UID: \"d7624aaf-8c64-4315-b1e2-635697aa8820\") " Oct 31 20:15:03.009601 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.008254 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca" pod "d7624aaf-8c64-4315-b1e2-635697aa8820" (UID: "d7624aaf-8c64-4315-b1e2-635697aa8820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:03.012018 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.011989 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca" pod "d7624aaf-8c64-4315-b1e2-635697aa8820" (UID: "d7624aaf-8c64-4315-b1e2-635697aa8820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:03.013515 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.013491 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb" pod "d7624aaf-8c64-4315-b1e2-635697aa8820" (UID: "d7624aaf-8c64-4315-b1e2-635697aa8820"). InnerVolumeSpecName "kube-api-access-fkwpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:03.109927 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.109840 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:03.109927 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.109879 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwpb\" (UniqueName: \"kubernetes.io/projected/d7624aaf-8c64-4315-b1e2-635697aa8820-kube-api-access-fkwpb\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:03.109927 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.109897 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7624aaf-8c64-4315-b1e2-635697aa8820-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:03.701667 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.700797 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" event={"ID":"d7624aaf-8c64-4315-b1e2-635697aa8820","Type":"ContainerDied","Data":"881347cdb661ebf5c77008263d586a7a27c8a6daca2805d79b91fe71c495f0c2"} Oct 31 20:15:03.701667 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.700856 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881347cdb661ebf5c77008263d586a7a27c8a6daca2805d79b91fe71c495f0c2" Oct 31 20:15:03.701667 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:03.700922 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512a862-hrfm5" Oct 31 20:15:10.425924 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.425890 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c"] Oct 31 20:15:10.426324 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.426279 5688 cpu_manager.go:414] "containerMap: removing container" podUID="d7624aaf-8c64-4315-b1e2-635697aa8820" containerName="revoke-keys" Oct 31 20:15:10.426324 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.426291 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7624aaf-8c64-4315-b1e2-635697aa8820" containerName="revoke-keys" Oct 31 20:15:10.426400 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.426359 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7624aaf-8c64-4315-b1e2-635697aa8820" containerName="revoke-keys" Oct 31 20:15:10.429454 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.429438 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.469029 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.469001 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.469146 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.469038 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.469146 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.469126 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwr4g\" (UniqueName: \"kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.481847 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.481828 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:15:10.481847 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.481834 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:15:10.481988 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.481863 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:15:10.531464 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.531442 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:15:10.533500 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.533477 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c"] Oct 31 20:15:10.570352 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.570330 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.570484 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.570406 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwr4g\" (UniqueName: \"kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.570550 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.570509 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.571054 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.571032 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.572687 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.572669 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.628141 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.628105 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwr4g\" (UniqueName: \"kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g\") pod \"maas-api-revoke-sub-12910354-w9b7c\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:10.741785 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:10.741763 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:11.113490 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:11.113465 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c"] Oct 31 20:15:11.115963 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:15:11.115933 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d97001_4417_44b3_8ba2_80c138554ed0.slice/crio-cbe377098eb211f5f27bda3914da31009537f765dbec54000cffd0e0a71ca305 WatchSource:0}: Error finding container cbe377098eb211f5f27bda3914da31009537f765dbec54000cffd0e0a71ca305: Status 404 returned error can't find the container with id cbe377098eb211f5f27bda3914da31009537f765dbec54000cffd0e0a71ca305 Oct 31 20:15:11.740383 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:11.740353 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" event={"ID":"63d97001-4417-44b3-8ba2-80c138554ed0","Type":"ContainerStarted","Data":"32629b0e4f5f565ad87f6638f1f528c502e42aa3d4f18d7d84a1c5a14c4512d5"} Oct 31 20:15:11.740755 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:11.740406 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" event={"ID":"63d97001-4417-44b3-8ba2-80c138554ed0","Type":"ContainerStarted","Data":"cbe377098eb211f5f27bda3914da31009537f765dbec54000cffd0e0a71ca305"} Oct 31 20:15:11.855430 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:11.855380 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" podStartSLOduration=1.85536666 podStartE2EDuration="1.85536666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:11.854415636 +0000 UTC m=+6832.456556779" watchObservedRunningTime="2026-10-31 20:15:11.85536666 +0000 UTC m=+6832.457507804" Oct 31 20:15:12.747070 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:12.747038 5688 generic.go:356] "Generic (PLEG): container finished" podID="63d97001-4417-44b3-8ba2-80c138554ed0" containerID="32629b0e4f5f565ad87f6638f1f528c502e42aa3d4f18d7d84a1c5a14c4512d5" exitCode=0 Oct 31 20:15:12.747476 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:12.747111 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" event={"ID":"63d97001-4417-44b3-8ba2-80c138554ed0","Type":"ContainerDied","Data":"32629b0e4f5f565ad87f6638f1f528c502e42aa3d4f18d7d84a1c5a14c4512d5"} Oct 31 20:15:14.147534 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.147345 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:14.215419 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.214697 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca\") pod \"63d97001-4417-44b3-8ba2-80c138554ed0\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " Oct 31 20:15:14.215419 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.214774 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca\") pod \"63d97001-4417-44b3-8ba2-80c138554ed0\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " Oct 31 20:15:14.215419 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.214826 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g\" (UniqueName: \"kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g\") pod \"63d97001-4417-44b3-8ba2-80c138554ed0\" (UID: \"63d97001-4417-44b3-8ba2-80c138554ed0\") " Oct 31 20:15:14.215714 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.215468 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca" pod "63d97001-4417-44b3-8ba2-80c138554ed0" (UID: "63d97001-4417-44b3-8ba2-80c138554ed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:14.217324 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.217300 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca" pod "63d97001-4417-44b3-8ba2-80c138554ed0" (UID: "63d97001-4417-44b3-8ba2-80c138554ed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:14.217665 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.217638 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g" pod "63d97001-4417-44b3-8ba2-80c138554ed0" (UID: "63d97001-4417-44b3-8ba2-80c138554ed0"). InnerVolumeSpecName "kube-api-access-kwr4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:14.315950 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.315916 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:14.315950 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.315945 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d97001-4417-44b3-8ba2-80c138554ed0-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:14.316146 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.315968 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwr4g\" (UniqueName: \"kubernetes.io/projected/63d97001-4417-44b3-8ba2-80c138554ed0-kube-api-access-kwr4g\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:14.801215 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.797463 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" event={"ID":"63d97001-4417-44b3-8ba2-80c138554ed0","Type":"ContainerDied","Data":"cbe377098eb211f5f27bda3914da31009537f765dbec54000cffd0e0a71ca305"} Oct 31 20:15:14.801215 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.797539 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe377098eb211f5f27bda3914da31009537f765dbec54000cffd0e0a71ca305" Oct 31 20:15:14.801215 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:14.797613 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12910354-w9b7c" Oct 31 20:15:21.228656 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.228622 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr"] Oct 31 20:15:21.229167 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.228961 5688 cpu_manager.go:414] "containerMap: removing container" podUID="63d97001-4417-44b3-8ba2-80c138554ed0" containerName="revoke-keys" Oct 31 20:15:21.229167 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.228971 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d97001-4417-44b3-8ba2-80c138554ed0" containerName="revoke-keys" Oct 31 20:15:21.229167 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.229040 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d97001-4417-44b3-8ba2-80c138554ed0" containerName="revoke-keys" Oct 31 20:15:21.232227 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.232206 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.269282 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.269254 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprnn\" (UniqueName: \"kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.269405 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.269300 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.269405 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.269325 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.284764 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.284742 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:15:21.334245 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.334214 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:15:21.334389 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.334292 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:15:21.334389 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.334316 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:15:21.336435 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.336410 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr"] Oct 31 20:15:21.369827 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.369793 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.369934 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.369837 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.369996 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.369944 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprnn\" (UniqueName: \"kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.370521 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.370498 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.372331 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.372310 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.426801 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.426777 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprnn\" (UniqueName: \"kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn\") pod \"maas-api-revoke-sub-b5c061e5-995lr\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.546205 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.546164 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:21.918223 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:21.918198 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr"] Oct 31 20:15:21.920224 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:15:21.920177 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22afafbf_888c_4547_b250_5fb216f8036a.slice/crio-15576f41f44b00208ba9364fc7b3a3400eff3b990bbf3932394c967d94ece27a WatchSource:0}: Error finding container 15576f41f44b00208ba9364fc7b3a3400eff3b990bbf3932394c967d94ece27a: Status 404 returned error can't find the container with id 15576f41f44b00208ba9364fc7b3a3400eff3b990bbf3932394c967d94ece27a Oct 31 20:15:22.836845 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:22.836813 5688 generic.go:356] "Generic (PLEG): container finished" podID="22afafbf-888c-4547-b250-5fb216f8036a" containerID="bab0c66aa9ab494b80c7bd8dd3b10f4ac8d25c20e9a1fac9ba3e9329b80b4e0f" exitCode=0 Oct 31 20:15:22.837168 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:22.836887 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" event={"ID":"22afafbf-888c-4547-b250-5fb216f8036a","Type":"ContainerDied","Data":"bab0c66aa9ab494b80c7bd8dd3b10f4ac8d25c20e9a1fac9ba3e9329b80b4e0f"} Oct 31 20:15:22.837168 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:22.836942 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" event={"ID":"22afafbf-888c-4547-b250-5fb216f8036a","Type":"ContainerStarted","Data":"15576f41f44b00208ba9364fc7b3a3400eff3b990bbf3932394c967d94ece27a"} Oct 31 20:15:23.978652 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:23.978613 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:24.083689 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.083653 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd"] Oct 31 20:15:24.086321 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.086297 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436da9be-jwcqd"] Oct 31 20:15:24.095033 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.094964 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca\") pod \"22afafbf-888c-4547-b250-5fb216f8036a\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " Oct 31 20:15:24.095033 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.095021 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca\") pod \"22afafbf-888c-4547-b250-5fb216f8036a\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " Oct 31 20:15:24.095342 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.095064 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn\" (UniqueName: \"kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn\") pod \"22afafbf-888c-4547-b250-5fb216f8036a\" (UID: \"22afafbf-888c-4547-b250-5fb216f8036a\") " Oct 31 20:15:24.097439 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.097394 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca" pod "22afafbf-888c-4547-b250-5fb216f8036a" (UID: "22afafbf-888c-4547-b250-5fb216f8036a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:24.099328 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.099302 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn" pod "22afafbf-888c-4547-b250-5fb216f8036a" (UID: "22afafbf-888c-4547-b250-5fb216f8036a"). InnerVolumeSpecName "kube-api-access-xprnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:24.101591 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.101566 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca" pod "22afafbf-888c-4547-b250-5fb216f8036a" (UID: "22afafbf-888c-4547-b250-5fb216f8036a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:24.196640 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.196610 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:24.196640 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.196633 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afafbf-888c-4547-b250-5fb216f8036a-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:24.196640 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.196643 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprnn\" (UniqueName: \"kubernetes.io/projected/22afafbf-888c-4547-b250-5fb216f8036a-kube-api-access-xprnn\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:24.875207 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.871597 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" event={"ID":"22afafbf-888c-4547-b250-5fb216f8036a","Type":"ContainerDied","Data":"15576f41f44b00208ba9364fc7b3a3400eff3b990bbf3932394c967d94ece27a"} Oct 31 20:15:24.875207 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.871658 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15576f41f44b00208ba9364fc7b3a3400eff3b990bbf3932394c967d94ece27a" Oct 31 20:15:24.875207 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:24.871725 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c061e5-995lr" Oct 31 20:15:24.927348 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:15:24.927315 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:15:25.576787 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:25.576751 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dd699e-7f86-4f71-9ac8-916aa4919973" path="/var/lib/kubelet/pods/80dd699e-7f86-4f71-9ac8-916aa4919973/volumes" Oct 31 20:15:32.012556 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.012474 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw"] Oct 31 20:15:32.012989 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.012802 5688 cpu_manager.go:414] "containerMap: removing container" podUID="22afafbf-888c-4547-b250-5fb216f8036a" containerName="revoke-keys" Oct 31 20:15:32.012989 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.012812 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22afafbf-888c-4547-b250-5fb216f8036a" containerName="revoke-keys" Oct 31 20:15:32.012989 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.012881 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="22afafbf-888c-4547-b250-5fb216f8036a" containerName="revoke-keys" Oct 31 20:15:32.017077 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.017059 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.056083 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.056046 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.056083 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.056080 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmzw\" (UniqueName: \"kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.056304 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.056246 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.073206 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.073153 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw"] Oct 31 20:15:32.120776 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.120750 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:15:32.120776 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.120773 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:15:32.120939 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.120779 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:15:32.120939 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.120819 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:15:32.156606 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.156579 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.156719 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.156612 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmzw\" (UniqueName: \"kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.156719 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.156664 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.157249 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.157232 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.159063 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.159043 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.214425 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.214398 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmzw\" (UniqueName: \"kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw\") pod \"maas-api-revoke-sub-d327d85c-2lmlw\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.329716 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.329660 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:32.706194 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.706158 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw"] Oct 31 20:15:32.708706 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:15:32.708678 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40c0602_70c4_4e1c_a29f_126ea4459eef.slice/crio-9d217e229a04f597a761ce11e696f920c3bc1c4a4c424f295f243bec00d04b83 WatchSource:0}: Error finding container 9d217e229a04f597a761ce11e696f920c3bc1c4a4c424f295f243bec00d04b83: Status 404 returned error can't find the container with id 9d217e229a04f597a761ce11e696f920c3bc1c4a4c424f295f243bec00d04b83 Oct 31 20:15:32.928361 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:32.928328 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" event={"ID":"b40c0602-70c4-4e1c-a29f-126ea4459eef","Type":"ContainerStarted","Data":"9d217e229a04f597a761ce11e696f920c3bc1c4a4c424f295f243bec00d04b83"} Oct 31 20:15:33.934652 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:33.934576 5688 generic.go:356] "Generic (PLEG): container finished" podID="b40c0602-70c4-4e1c-a29f-126ea4459eef" containerID="e01d36a7a9b8bf4d1bc0883888e8f8bfe3c4f7f70dd6d56a3f0520a102037d36" exitCode=0 Oct 31 20:15:33.934996 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:33.934660 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" event={"ID":"b40c0602-70c4-4e1c-a29f-126ea4459eef","Type":"ContainerDied","Data":"e01d36a7a9b8bf4d1bc0883888e8f8bfe3c4f7f70dd6d56a3f0520a102037d36"} Oct 31 20:15:35.049337 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.049298 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78"] Oct 31 20:15:35.053386 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.053361 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf94c23-j8k78"] Oct 31 20:15:35.093264 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.093243 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:35.177698 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.177670 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw\" (UniqueName: \"kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw\") pod \"b40c0602-70c4-4e1c-a29f-126ea4459eef\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " Oct 31 20:15:35.177837 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.177725 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca\") pod \"b40c0602-70c4-4e1c-a29f-126ea4459eef\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " Oct 31 20:15:35.177935 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.177915 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca\") pod \"b40c0602-70c4-4e1c-a29f-126ea4459eef\" (UID: \"b40c0602-70c4-4e1c-a29f-126ea4459eef\") " Oct 31 20:15:35.178407 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.178331 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca" pod "b40c0602-70c4-4e1c-a29f-126ea4459eef" (UID: "b40c0602-70c4-4e1c-a29f-126ea4459eef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:35.179880 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.179852 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca" pod "b40c0602-70c4-4e1c-a29f-126ea4459eef" (UID: "b40c0602-70c4-4e1c-a29f-126ea4459eef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:35.180373 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.180352 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw" pod "b40c0602-70c4-4e1c-a29f-126ea4459eef" (UID: "b40c0602-70c4-4e1c-a29f-126ea4459eef"). InnerVolumeSpecName "kube-api-access-wxmzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:35.278986 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.278949 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxmzw\" (UniqueName: \"kubernetes.io/projected/b40c0602-70c4-4e1c-a29f-126ea4459eef-kube-api-access-wxmzw\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:35.278986 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.278985 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:35.279273 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.279002 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40c0602-70c4-4e1c-a29f-126ea4459eef-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:35.583448 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.583365 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6da421-b9ab-475f-a6b0-0cc4263598fd" path="/var/lib/kubelet/pods/6b6da421-b9ab-475f-a6b0-0cc4263598fd/volumes" Oct 31 20:15:35.965262 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.964384 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" event={"ID":"b40c0602-70c4-4e1c-a29f-126ea4459eef","Type":"ContainerDied","Data":"9d217e229a04f597a761ce11e696f920c3bc1c4a4c424f295f243bec00d04b83"} Oct 31 20:15:35.965262 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.964488 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d217e229a04f597a761ce11e696f920c3bc1c4a4c424f295f243bec00d04b83" Oct 31 20:15:35.965262 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:35.964524 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d327d85c-2lmlw" Oct 31 20:15:42.766262 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.766233 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6"] Oct 31 20:15:42.766662 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.766572 5688 cpu_manager.go:414] "containerMap: removing container" podUID="b40c0602-70c4-4e1c-a29f-126ea4459eef" containerName="revoke-keys" Oct 31 20:15:42.766662 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.766582 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40c0602-70c4-4e1c-a29f-126ea4459eef" containerName="revoke-keys" Oct 31 20:15:42.766662 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.766643 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40c0602-70c4-4e1c-a29f-126ea4459eef" containerName="revoke-keys" Oct 31 20:15:42.770570 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.770544 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.850591 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.850561 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.850762 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.850598 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.850762 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.850694 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcj6\" (UniqueName: \"kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.875700 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.875678 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:15:42.875700 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.875680 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:15:42.875700 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.875684 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:15:42.875888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.875722 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:15:42.877879 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.877862 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6"] Oct 31 20:15:42.951055 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.951032 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcj6\" (UniqueName: \"kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.951160 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.951089 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.951160 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.951122 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.951757 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.951736 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:42.953495 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:42.953471 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:43.008514 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:43.008491 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcj6\" (UniqueName: \"kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6\") pod \"maas-api-revoke-sub-e74c6f34-5t7h6\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:43.086741 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:43.086692 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:43.459808 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:43.459788 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6"] Oct 31 20:15:43.461816 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:15:43.461784 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b7bc08_db9e_4e3e_a8c1_846f4fd734e4.slice/crio-8ccb24f9df8c190b451b0c99effbbc4fbddf16378da883f0e27dd155df3932bd WatchSource:0}: Error finding container 8ccb24f9df8c190b451b0c99effbbc4fbddf16378da883f0e27dd155df3932bd: Status 404 returned error can't find the container with id 8ccb24f9df8c190b451b0c99effbbc4fbddf16378da883f0e27dd155df3932bd Oct 31 20:15:44.007455 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:44.007425 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" event={"ID":"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4","Type":"ContainerStarted","Data":"53059fa0b0fdc773f0dc2807e6a02ac49bd08356eb4f6e29898ed37c292db144"} Oct 31 20:15:44.007880 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:44.007470 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" event={"ID":"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4","Type":"ContainerStarted","Data":"8ccb24f9df8c190b451b0c99effbbc4fbddf16378da883f0e27dd155df3932bd"} Oct 31 20:15:44.123437 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:44.123392 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" podStartSLOduration=2.123378829 podStartE2EDuration="2.123378829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:44.122313387 +0000 UTC m=+6864.724454532" watchObservedRunningTime="2026-10-31 20:15:44.123378829 +0000 UTC m=+6864.725519973" Oct 31 20:15:45.013438 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:45.013410 5688 generic.go:356] "Generic (PLEG): container finished" podID="c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" containerID="53059fa0b0fdc773f0dc2807e6a02ac49bd08356eb4f6e29898ed37c292db144" exitCode=0 Oct 31 20:15:45.013813 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:45.013500 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" event={"ID":"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4","Type":"ContainerDied","Data":"53059fa0b0fdc773f0dc2807e6a02ac49bd08356eb4f6e29898ed37c292db144"} Oct 31 20:15:46.165096 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.165068 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:46.182432 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.182392 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6\" (UniqueName: \"kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6\") pod \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " Oct 31 20:15:46.183065 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.183045 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca\") pod \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " Oct 31 20:15:46.194380 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.194355 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca\") pod \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\" (UID: \"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4\") " Oct 31 20:15:46.194976 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.194851 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca" pod "c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" (UID: "c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:46.195411 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.195383 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:46.222408 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.221487 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca" pod "c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" (UID: "c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:46.223376 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.223340 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6" pod "c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" (UID: "c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4"). InnerVolumeSpecName "kube-api-access-9fcj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:46.299249 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.298677 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:46.299249 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:46.298711 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcj6\" (UniqueName: \"kubernetes.io/projected/c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4-kube-api-access-9fcj6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:47.026237 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:47.026208 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" Oct 31 20:15:47.026411 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:47.026232 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74c6f34-5t7h6" event={"ID":"c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4","Type":"ContainerDied","Data":"8ccb24f9df8c190b451b0c99effbbc4fbddf16378da883f0e27dd155df3932bd"} Oct 31 20:15:47.026411 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:47.026284 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ccb24f9df8c190b451b0c99effbbc4fbddf16378da883f0e27dd155df3932bd" Oct 31 20:15:53.568623 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.568594 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf"] Oct 31 20:15:53.569110 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.568932 5688 cpu_manager.go:414] "containerMap: removing container" podUID="c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" containerName="revoke-keys" Oct 31 20:15:53.569110 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.568947 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" containerName="revoke-keys" Oct 31 20:15:53.569110 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.569033 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4" containerName="revoke-keys" Oct 31 20:15:53.572376 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.572353 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.627865 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.627840 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\"" Oct 31 20:15:53.628003 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.627976 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:15:53.667527 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.667500 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.667660 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.667533 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szz6\" (UniqueName: \"kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.667660 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.667570 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.677670 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.677651 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:15:53.677670 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.677661 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:15:53.679753 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.679735 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf"] Oct 31 20:15:53.768813 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.768784 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.768939 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.768824 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szz6\" (UniqueName: \"kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.768939 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.768888 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.769494 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.769472 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.771316 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.771299 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.825340 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.825288 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szz6\" (UniqueName: \"kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6\") pod \"maas-api-revoke-sub-aa30db0a-lx6hf\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:53.885373 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:53.885351 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:54.253775 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:54.253752 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf"] Oct 31 20:15:54.256035 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:15:54.256010 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c2f3d8_4441_4e2b_b096_ae47e6869dd6.slice/crio-e0836af8e691eb87860c183453c20f02773f67a3bf7a48f1cc92ed59e690dfa7 WatchSource:0}: Error finding container e0836af8e691eb87860c183453c20f02773f67a3bf7a48f1cc92ed59e690dfa7: Status 404 returned error can't find the container with id e0836af8e691eb87860c183453c20f02773f67a3bf7a48f1cc92ed59e690dfa7 Oct 31 20:15:54.902694 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:15:54.902665 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:15:55.065216 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:55.065170 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" event={"ID":"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6","Type":"ContainerStarted","Data":"277a2809365271b770c7229b26bec03173f75fd89d3907b7a40271a54831ec7d"} Oct 31 20:15:55.065403 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:55.065239 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" event={"ID":"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6","Type":"ContainerStarted","Data":"e0836af8e691eb87860c183453c20f02773f67a3bf7a48f1cc92ed59e690dfa7"} Oct 31 20:15:55.180724 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:55.180632 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" podStartSLOduration=2.180611294 podStartE2EDuration="2.180611294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:55.179403709 +0000 UTC m=+6875.781544852" watchObservedRunningTime="2026-10-31 20:15:55.180611294 +0000 UTC m=+6875.782752436" Oct 31 20:15:56.073783 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:56.073746 5688 generic.go:356] "Generic (PLEG): container finished" podID="e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" containerID="277a2809365271b770c7229b26bec03173f75fd89d3907b7a40271a54831ec7d" exitCode=0 Oct 31 20:15:56.074240 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:56.073833 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" event={"ID":"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6","Type":"ContainerDied","Data":"277a2809365271b770c7229b26bec03173f75fd89d3907b7a40271a54831ec7d"} Oct 31 20:15:57.420376 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.420352 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:57.515588 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.515563 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6\" (UniqueName: \"kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6\") pod \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " Oct 31 20:15:57.515714 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.515605 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca\") pod \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " Oct 31 20:15:57.515714 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.515658 5688 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca\") pod \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\" (UID: \"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6\") " Oct 31 20:15:57.516018 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.515993 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca" pod "e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" (UID: "e4c2f3d8-4441-4e2b-b096-ae47e6869dd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:57.517672 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.517655 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca" pod "e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" (UID: "e4c2f3d8-4441-4e2b-b096-ae47e6869dd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:57.518581 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.518564 5688 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6" pod "e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" (UID: "e4c2f3d8-4441-4e2b-b096-ae47e6869dd6"). InnerVolumeSpecName "kube-api-access-9szz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:57.616459 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.616409 5688 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9szz6\" (UniqueName: \"kubernetes.io/projected/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-kube-api-access-9szz6\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:57.616459 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.616444 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-openshift-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:57.616459 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:57.616461 5688 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c2f3d8-4441-4e2b-b096-ae47e6869dd6-maas-api-xks-ca\") on node \"ip-10-0-140-47.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 20:15:58.102126 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:58.102096 5688 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" Oct 31 20:15:58.102338 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:58.102141 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa30db0a-lx6hf" event={"ID":"e4c2f3d8-4441-4e2b-b096-ae47e6869dd6","Type":"ContainerDied","Data":"e0836af8e691eb87860c183453c20f02773f67a3bf7a48f1cc92ed59e690dfa7"} Oct 31 20:15:58.102338 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:15:58.102210 5688 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0836af8e691eb87860c183453c20f02773f67a3bf7a48f1cc92ed59e690dfa7" Oct 31 20:16:11.053525 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:11.053458 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs"] Oct 31 20:16:11.103175 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:11.103141 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829b426d-d7lrs"] Oct 31 20:16:11.564080 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:11.564048 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f87461-a7b7-4d97-b76c-b8053d252060" path="/var/lib/kubelet/pods/a2f87461-a7b7-4d97-b76c-b8053d252060/volumes" Oct 31 20:16:20.470696 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:20.470665 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:16:20.482460 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:20.482439 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zqcnd_12259580-c199-4cac-a5c2-35a79e28ca90/console-operator/2.log" Oct 31 20:16:22.568129 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:22.568098 5688 scope.go:122] "RemoveContainer" containerID="632cf9cb27bc55c28361951b46d326906466f14c46a4e41f0256e93546a3fd68" Oct 31 20:16:24.903014 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:16:24.902984 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:16:38.654345 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:38.654319 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-v99xr_17f0ac62-e7aa-4dea-8bae-dede767972e6/manager/0.log" Oct 31 20:16:39.764483 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:39.764436 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cgxbd_c75c1ba8-2b81-4a05-bf20-22aea0bdc837/manager/0.log" Oct 31 20:16:42.056099 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:42.056067 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-058ab0b2-dlln4_4f4e8554-7182-4fe6-b503-c78653b319d2/revoke-keys/0.log" Oct 31 20:16:42.287731 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:42.287706 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0e6d39b1-knm7r_3b5f96c4-67a5-493a-9c10-5966f2af8d61/revoke-keys/0.log" Oct 31 20:16:42.504754 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:42.504734 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-12910354-w9b7c_63d97001-4417-44b3-8ba2-80c138554ed0/revoke-keys/0.log" Oct 31 20:16:42.732105 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:42.732084 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-44fa7a57-2xrgq_8a2baaab-af86-420b-89cc-7c7c4195e0eb/revoke-keys/0.log" Oct 31 20:16:42.957415 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:42.957361 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5427ae91-86vmj_9df9228b-5cbb-4c79-9070-8663cb71098a/revoke-keys/0.log" Oct 31 20:16:43.185935 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:43.185914 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6512a862-hrfm5_d7624aaf-8c64-4315-b1e2-635697aa8820/revoke-keys/0.log" Oct 31 20:16:43.404978 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:43.404953 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8292ee71-648tv_6422ed4b-0593-4817-bde3-d7fb7f648bb2/revoke-keys/0.log" Oct 31 20:16:43.621483 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:43.621456 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9ce7568e-nmlng_e9a574ff-526d-4931-9d9f-07fdfb13f9ec/revoke-keys/0.log" Oct 31 20:16:43.848792 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:43.848770 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a8d0e5cc-nb5td_56426894-02f1-43e8-8fec-407765c009ab/revoke-keys/0.log" Oct 31 20:16:44.071716 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:44.071690 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aa30db0a-lx6hf_e4c2f3d8-4441-4e2b-b096-ae47e6869dd6/revoke-keys/0.log" Oct 31 20:16:44.308069 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:44.308047 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b5c061e5-995lr_22afafbf-888c-4547-b250-5fb216f8036a/revoke-keys/0.log" Oct 31 20:16:44.537073 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:44.537051 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3d0d789-4jqkb_58e58c4f-3312-4f27-957f-a4e46f856954/revoke-keys/0.log" Oct 31 20:16:44.754047 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:44.754023 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d327d85c-2lmlw_b40c0602-70c4-4e1c-a29f-126ea4459eef/revoke-keys/0.log" Oct 31 20:16:44.980429 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:44.980403 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d65e62dd-q8mng_c30ad12a-d428-4567-9f73-002e332c9dbd/revoke-keys/0.log" Oct 31 20:16:45.208646 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:45.208585 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e74c6f34-5t7h6_c4b7bc08-db9e-4e3e-a8c1-846f4fd734e4/revoke-keys/0.log" Oct 31 20:16:45.434708 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:45.434686 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f18900b6-wgdxc_c77b78d6-f806-4700-a349-4427a0fb1025/revoke-keys/0.log" Oct 31 20:16:48.540798 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:48.540764 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4ddzf_5c4762c4-658b-4a35-bfc5-b89a29bb2e11/kuadrant-console-plugin/0.log" Oct 31 20:16:51.027489 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:51.027447 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mlhcv_16e4ddc8-e42c-46d6-8d9b-8784b06d4760/discovery/0.log" Oct 31 20:16:51.263827 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:51.263797 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4f97d94f-8g88g_8a491c77-ee30-48ea-b10f-8f700ff7d850/kube-auth-proxy/0.log" Oct 31 20:16:51.719585 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:51.719553 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-85vdx_3affcc65-df38-4880-8581-75f2dffc6b50/istio-proxy/0.log" Oct 31 20:16:52.443436 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:52.443412 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-cp75p_ea3982da-e2ec-4ffb-9b3f-a805c03bcbab/istio-proxy/0.log" Oct 31 20:16:53.987466 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:53.987441 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-ztb8k_b1c9fc38-2e60-4b56-a5dd-a223912fc013/istio-proxy/0.log" Oct 31 20:16:54.903366 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:16:54.903335 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:16:54.937566 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:54.937536 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59bdcd5559-qvgdn_f17f5606-10aa-4f02-a2c4-16c28aba8540/router/0.log" Oct 31 20:16:56.045252 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:56.045219 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td"] Oct 31 20:16:56.094698 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:56.094672 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0e5cc-nb5td"] Oct 31 20:16:57.563719 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:16:57.563686 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56426894-02f1-43e8-8fec-407765c009ab" path="/var/lib/kubelet/pods/56426894-02f1-43e8-8fec-407765c009ab/volumes" Oct 31 20:17:07.051964 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.051927 5688 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq"] Oct 31 20:17:07.101315 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.101285 5688 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44fa7a57-2xrgq"] Oct 31 20:17:07.564527 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.564479 5688 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2baaab-af86-420b-89cc-7c7c4195e0eb" path="/var/lib/kubelet/pods/8a2baaab-af86-420b-89cc-7c7c4195e0eb/volumes" Oct 31 20:17:07.951254 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.951174 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bxfnk/must-gather-pzz8k"] Oct 31 20:17:07.951553 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.951539 5688 cpu_manager.go:414] "containerMap: removing container" podUID="e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" containerName="revoke-keys" Oct 31 20:17:07.951613 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.951554 5688 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" containerName="revoke-keys" Oct 31 20:17:07.951657 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.951649 5688 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c2f3d8-4441-4e2b-b096-ae47e6869dd6" containerName="revoke-keys" Oct 31 20:17:07.955090 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:07.955073 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.012553 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.012528 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxfnk\"/\"openshift-service-ca.crt\"" Oct 31 20:17:08.060080 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.060059 5688 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxfnk\"/\"kube-root-ca.crt\"" Oct 31 20:17:08.060448 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.060423 5688 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bxfnk\"/\"default-dockercfg-l2c59\"" Oct 31 20:17:08.062515 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.062496 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/must-gather-pzz8k"] Oct 31 20:17:08.068762 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.068744 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lw9v\" (UniqueName: \"kubernetes.io/projected/8cfe7f53-14dc-406a-95b7-11f0088841ce-kube-api-access-8lw9v\") pod \"must-gather-pzz8k\" (UID: \"8cfe7f53-14dc-406a-95b7-11f0088841ce\") " pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.068856 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.068789 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8cfe7f53-14dc-406a-95b7-11f0088841ce-must-gather-output\") pod \"must-gather-pzz8k\" (UID: \"8cfe7f53-14dc-406a-95b7-11f0088841ce\") " pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.169467 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.169442 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8cfe7f53-14dc-406a-95b7-11f0088841ce-must-gather-output\") pod \"must-gather-pzz8k\" (UID: \"8cfe7f53-14dc-406a-95b7-11f0088841ce\") " pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.169565 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.169496 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lw9v\" (UniqueName: \"kubernetes.io/projected/8cfe7f53-14dc-406a-95b7-11f0088841ce-kube-api-access-8lw9v\") pod \"must-gather-pzz8k\" (UID: \"8cfe7f53-14dc-406a-95b7-11f0088841ce\") " pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.169759 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.169740 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8cfe7f53-14dc-406a-95b7-11f0088841ce-must-gather-output\") pod \"must-gather-pzz8k\" (UID: \"8cfe7f53-14dc-406a-95b7-11f0088841ce\") " pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.227698 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.227647 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lw9v\" (UniqueName: \"kubernetes.io/projected/8cfe7f53-14dc-406a-95b7-11f0088841ce-kube-api-access-8lw9v\") pod \"must-gather-pzz8k\" (UID: \"8cfe7f53-14dc-406a-95b7-11f0088841ce\") " pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.268556 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.268534 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/must-gather-pzz8k" Oct 31 20:17:08.640367 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.640314 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/must-gather-pzz8k"] Oct 31 20:17:08.642624 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:17:08.642593 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cfe7f53_14dc_406a_95b7_11f0088841ce.slice/crio-c1722345455b71b6a703aa60481f3d0cef96729b300d3d6e7927a34c11f0f218 WatchSource:0}: Error finding container c1722345455b71b6a703aa60481f3d0cef96729b300d3d6e7927a34c11f0f218: Status 404 returned error can't find the container with id c1722345455b71b6a703aa60481f3d0cef96729b300d3d6e7927a34c11f0f218 Oct 31 20:17:08.644315 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:08.644302 5688 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:17:09.481175 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:09.481141 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/must-gather-pzz8k" event={"ID":"8cfe7f53-14dc-406a-95b7-11f0088841ce","Type":"ContainerStarted","Data":"c1722345455b71b6a703aa60481f3d0cef96729b300d3d6e7927a34c11f0f218"} Oct 31 20:17:10.487898 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:10.487862 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/must-gather-pzz8k" event={"ID":"8cfe7f53-14dc-406a-95b7-11f0088841ce","Type":"ContainerStarted","Data":"a421c222bdc01d7508918a7797941cdb5e6bde645be28351c288b348e4093741"} Oct 31 20:17:10.488297 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:10.487902 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/must-gather-pzz8k" event={"ID":"8cfe7f53-14dc-406a-95b7-11f0088841ce","Type":"ContainerStarted","Data":"31634281f8c824064c725bf7afef696a49b97974f3cff10ded79405ce5f04305"} Oct 31 20:17:10.601595 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:10.601548 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bxfnk/must-gather-pzz8k" podStartSLOduration=2.215756164 podStartE2EDuration="3.601533412s" totalImagesPullingTime="1.385777248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:17:10.601271172 +0000 UTC m=+6951.203412315" watchObservedRunningTime="2026-10-31 20:17:10.601533412 +0000 UTC m=+6951.203674556" Oct 31 20:17:18.089172 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:18.087686 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tczvl_96e03cbb-2909-4802-accd-002dd2922c42/global-pull-secret-syncer/0.log" Oct 31 20:17:18.681654 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:18.681459 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ksnkf_4334dda9-cc0b-49bd-a5ca-48457df595fe/connection-checker/0.log" Oct 31 20:17:19.113454 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:19.113082 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-v4vbz_25825c7b-a2c6-4263-bc62-d2c85c00ec47/connection-checker/0.log" Oct 31 20:17:19.545274 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:19.545248 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xgpld_875a3d19-4e6c-4bb6-87b9-e801cd4f6604/connection-checker/0.log" Oct 31 20:17:21.008930 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:21.008892 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v2bs5_cc74a8fe-8053-4b78-b8bf-569e5e34a89f/konnectivity-agent/0.log" Oct 31 20:17:21.820653 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:21.820622 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-47.us-west-1.compute.internal_4d60099914d5eef84a51e7038f8e20de/haproxy/0.log" Oct 31 20:17:22.589487 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:22.589461 5688 scope.go:122] "RemoveContainer" containerID="c48ae16d7da4470057ecd76fc943cb4d6519ae4384aa9c2b57f0bc266101fe05" Oct 31 20:17:22.633484 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:22.633456 5688 scope.go:122] "RemoveContainer" containerID="aacf0edac90a3a0fcffdf5c922a91159228e09a0ea1dc546d23e92a62dea83ff" Oct 31 20:17:24.925132 ip-10-0-140-47 kubenswrapper[5688]: E1031 20:17:24.925100 5688 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-np8hr" containerName="extractor" Oct 31 20:17:26.779335 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.779286 5688 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42"] Oct 31 20:17:26.786214 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.786172 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:26.814647 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.814619 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-v99xr_17f0ac62-e7aa-4dea-8bae-dede767972e6/manager/0.log" Oct 31 20:17:26.844746 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.843173 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42"] Oct 31 20:17:26.931871 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.931838 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-proc\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:26.931871 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.931876 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-podres\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:26.932241 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.931905 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-lib-modules\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:26.932241 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.932046 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-sys\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:26.932241 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:26.932079 5688 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8dl\" (UniqueName: \"kubernetes.io/projected/37bd09d3-761d-4108-bab3-657b239fe141-kube-api-access-cs8dl\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032662 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032576 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-proc\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032662 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032612 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-podres\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032662 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032630 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-lib-modules\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032671 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-sys\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032689 5688 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8dl\" (UniqueName: \"kubernetes.io/projected/37bd09d3-761d-4108-bab3-657b239fe141-kube-api-access-cs8dl\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032707 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-podres\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032707 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-proc\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032758 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-lib-modules\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.032888 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.032767 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37bd09d3-761d-4108-bab3-657b239fe141-sys\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.090141 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.090106 5688 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8dl\" (UniqueName: \"kubernetes.io/projected/37bd09d3-761d-4108-bab3-657b239fe141-kube-api-access-cs8dl\") pod \"perf-node-gather-daemonset-j8j42\" (UID: \"37bd09d3-761d-4108-bab3-657b239fe141\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.102938 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.102909 5688 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.481084 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.480995 5688 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42"] Oct 31 20:17:27.485488 ip-10-0-140-47 kubenswrapper[5688]: W1031 20:17:27.485457 5688 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37bd09d3_761d_4108_bab3_657b239fe141.slice/crio-d57b4c8f0a24d9ba038c13a7456de119258e4205438bc979397f1f5e8c40aa2b WatchSource:0}: Error finding container d57b4c8f0a24d9ba038c13a7456de119258e4205438bc979397f1f5e8c40aa2b: Status 404 returned error can't find the container with id d57b4c8f0a24d9ba038c13a7456de119258e4205438bc979397f1f5e8c40aa2b Oct 31 20:17:27.595360 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.595312 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" event={"ID":"37bd09d3-761d-4108-bab3-657b239fe141","Type":"ContainerStarted","Data":"f8343a4262fbd6f6a1e275d0724b1d193128ab7407a04b8e1cb9e6dae8592927"} Oct 31 20:17:27.595546 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.595384 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:27.595546 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.595405 5688 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" event={"ID":"37bd09d3-761d-4108-bab3-657b239fe141","Type":"ContainerStarted","Data":"d57b4c8f0a24d9ba038c13a7456de119258e4205438bc979397f1f5e8c40aa2b"} Oct 31 20:17:27.633815 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.633784 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4ddzf_5c4762c4-658b-4a35-bfc5-b89a29bb2e11/kuadrant-console-plugin/0.log" Oct 31 20:17:27.710958 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:27.710534 5688 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" podStartSLOduration=1.710515585 podStartE2EDuration="1.710515585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:17:27.710453582 +0000 UTC m=+6968.312594728" watchObservedRunningTime="2026-10-31 20:17:27.710515585 +0000 UTC m=+6968.312656730" Oct 31 20:17:28.488631 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:28.488597 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cgxbd_c75c1ba8-2b81-4a05-bf20-22aea0bdc837/manager/0.log" Oct 31 20:17:32.678725 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:32.678688 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-v99xr_17f0ac62-e7aa-4dea-8bae-dede767972e6/manager/0.log" Oct 31 20:17:33.614422 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:33.614390 5688 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-j8j42" Oct 31 20:17:34.226224 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:34.224953 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cgxbd_c75c1ba8-2b81-4a05-bf20-22aea0bdc837/manager/0.log" Oct 31 20:17:34.614580 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:34.614547 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-wnxfz_52aa29ef-34be-481f-b3a6-d78bfeb0b181/cluster-monitoring-operator/0.log" Oct 31 20:17:38.407937 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:38.407900 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-v99xr_17f0ac62-e7aa-4dea-8bae-dede767972e6/manager/0.log" Oct 31 20:17:39.658302 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:39.658264 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r55xr_08606964-6c14-4183-ba74-4973e60e692b/node-exporter/0.log" Oct 31 20:17:40.035889 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:40.035858 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cgxbd_c75c1ba8-2b81-4a05-bf20-22aea0bdc837/manager/0.log" Oct 31 20:17:40.076526 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:40.076496 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r55xr_08606964-6c14-4183-ba74-4973e60e692b/kube-rbac-proxy/0.log" Oct 31 20:17:40.490647 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:40.490618 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r55xr_08606964-6c14-4183-ba74-4973e60e692b/init-textfile/0.log" Oct 31 20:17:45.144857 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:45.144822 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mlhcv_16e4ddc8-e42c-46d6-8d9b-8784b06d4760/discovery/0.log" Oct 31 20:17:45.268378 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:45.268352 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4f97d94f-8g88g_8a491c77-ee30-48ea-b10f-8f700ff7d850/kube-auth-proxy/0.log" Oct 31 20:17:45.713577 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:45.713550 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-85vdx_3affcc65-df38-4880-8581-75f2dffc6b50/istio-proxy/0.log" Oct 31 20:17:46.355732 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:46.355701 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-cp75p_ea3982da-e2ec-4ffb-9b3f-a805c03bcbab/istio-proxy/0.log" Oct 31 20:17:47.636738 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:47.636704 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-ztb8k_b1c9fc38-2e60-4b56-a5dd-a223912fc013/istio-proxy/0.log" Oct 31 20:17:48.487572 ip-10-0-140-47 kubenswrapper[5688]: I1031 20:17:48.487544 5688 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59bdcd5559-qvgdn_f17f5606-10aa-4f02-a2c4-16c28aba8540/router/0.log"